Sound signature database for initialization of noise reduction in recordings

ABSTRACT

A smart-home device may include a recording device configured to record sound during a first time interval and a memory device comprising a plurality of stored sound profiles. The smart-home device may also include a processing system configured to receive an environmental input, select a stored sound profile from the plurality of stored sound profiles based on the environmental input, and perform a noise-cancelation routine on the sound recorded during the first time interval. The stored sound profile may be used as an initial background noise profile for the noise-cancelation routine.

CROSS-REFERENCES TO RELATED APPLICATIONS

The subject matter of the instant disclosure is related to the subjectmatter of the following commonly assigned applications, each of which isincorporated by reference herein for all purposes: U.S. Ser. No.14/692,551 filed on Apr. 21, 2015, entitled “CUSTOMIZINGSPEECH-RECOGNITION DICTIONARIES IN A SMART-HOME ENVIRONMENT.”

BACKGROUND OF THE INVENTION

Some homes today are equipped with smart home networks to provideautomated control of devices, appliances and systems, such as heating,ventilation, and air conditioning (“HVAC”) systems, lighting systems,alarm systems, and home theater and entertainment systems. Smart homenetworks may include control panels that a person may use to inputsettings, preferences, and scheduling information that the smart homenetwork uses to provide automated control the various devices,appliances and systems in the home.

BRIEF SUMMARY OF THE INVENTION

In some embodiments, a smart-home device may include a recording deviceconfigured to record sound during a first time interval, and a memorydevice that includes a plurality of stored sound profiles. Thesmart-home device may also include a processing system configured toreceive an environmental input, select a stored sound profile from theplurality of stored sound profiles based on the environmental input, andperform a noise-cancelation routine on the sound recorded during thefirst time interval. In some embodiments, the stored sound profile maybe used as an initial background noise profile for the noise-cancelationroutine.

In some embodiments, a method of selecting an initial background noisefor a noise-cancelation routine in a smart-home device may includereceiving, by the smart-home device, an environmental input. The methodmay also include selecting, using a processor on the smart-home device,a stored sound profile from a plurality of stored sound profiles. Insome embodiments, the stored sound profile may be selected based on theenvironmental input. The method may additionally include performing thenoise-cancelation routine on sound received through a microphone of thesmart-home device. In some embodiments, the noise-cancelation routinemay be performed using the stored sound profile as an initial backgroundnoise for the noise-cancelation routine.

In some embodiments, a non-transitory memory device may be presented.The memory device may include instructions that, when executed by one ormore processors, cause the one or more processors to perform operationsincluding receiving, by the one or more processors, an environmentalinput. The operations may also include selecting, using the one or moreprocessors, a stored sound profile from a plurality of stored soundprofiles. In some embodiments, the stored sound profile may be selectedbased on the environmental input. The operations may additionallyinclude performing the noise-cancelation routine on a received sound. Insome embodiments, the noise-cancelation routine may be performed usingthe stored sound profile as an initial background noise for thenoise-cancelation routine.

In various implementations, one or more of the following features may beincorporated in any combination and without limitation. The smart-homedevice may include a hazard detector that includes a smoke detector anda carbon monoxide detector. The smart-home device may include athermostat that includes a temperature sensor; and a motion sensor. Thesmart-home device may include a network interface in communication witha remote server. The processing system may be further configured to:receive a second environmental input; send the second environmentalinput to a remote server through the network interface; receive a secondstored sound profile through the network interface from the remoteserver, where the remote server selected the second stored sound profilebased on the second environmental input; and perform thenoise-cancellation routine on sound recorded during a second timeinterval, where the second stored sound profile is used as an initialbackground noise profile for the noise-cancellation routine. Theprocessing system may be further configured to: record a new soundprofile using the microphone; and send the new sound profile to theremote server through the network interface. The smart-home device maybe in communication through the network interface with an appliancewithin the same home; and the environmental input may be received fromthe appliance. The stored sound profile may be recorded by a secondsmart-home device within the same home and transmitted from the secondsmart-home device to the smart-home device. The environmental input mayinclude a time of day. The environmental input may include a soundrecording. The environmental input may include an input received from ahousehold appliance. The sound received through the microphone of thesmart-home device may have a duration of less than approximately 10seconds. The noise-cancellation routine may use the initial backgroundnoise as a starting point for a convergence algorithm. Themethod/operations may also include determining that a second storedsound profile will cause the convergence algorithm to converge fasterthan the stored sound profile; and providing the second stored soundprofile to the noise-cancellation routine. The method/operations mayadditionally include during a learning interval, building the pluralityof stored sound profiles by detecting a time interval during which astudy-state background noise is present; recording the steady-statebackground noise; and recording characteristics of the occurrence of thestudy-state background noise. The stored sound profile may be associatedwith sound emitted by a household appliance during operation. Thereceived sound may include a voice command. The noise-cancellationroutine may be performed on the received sound in real-time.

To better understand the nature and advantages of the present invention,reference should be made to the following description and theaccompanying figures. It is to be understood, however, that each of thefigures is provided for the purpose of illustration only and is notintended as a definition of the limits of the scope of the presentinvention. Also, as a general rule, and unless it is evident to thecontrary from the description, where elements in different figures useidentical reference numbers, the elements are generally either identicalor at least similar in function or purpose.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a smart-home environment within which one ormore of the devices, methods, systems, services, and/or computer programproducts described further herein will be applicable, according to someembodiments.

FIG. 2 illustrates a network-level view of an extensible devices andservices platform with which the smart-home environment of FIG. 1 can beintegrated, according to some embodiments.

FIG. 3 illustrates an abstracted functional view of the extensibledevices and services platform of FIG. 2, with reference to a processingengine as well as devices of the smart-home environment, according tosome embodiments.

FIGS. 4A-4B illustrate perspective exploded and assembled views,respectively, of an intelligent, multi-sensing, network-connected hazarddetector, according to some embodiments.

FIGS. 5A-5B illustrate front and rear perspective views of a circuitboard of the hazard detector of FIGS. 4A-4B, according to someembodiments.

FIGS. 5C-5D illustrate front and rear perspective views of a speakerthat is mountable on the circuit board of the hazard detector of FIGS.4A-4B, according to some embodiments.

FIGS. 6A-6B illustrate front and rear perspective views of a lens buttonof the hazard detector of FIGS. 4A-4B, according to some embodiments.

FIGS. 6C-6D illustrate front and rear perspective views of a light guideof the hazard detector of FIGS. 4A-4B, according to some embodiments.

FIGS. 6E-6F illustrate front and rear perspective views of a flexiblestrip of the hazard detector of FIGS. 4A-4B, according to someembodiments.

FIG. 7 illustrates a block diagram of a smart-home device architecture,according to some embodiments.

FIG. 8 illustrates a flowchart of a method for building a database ofstored sound profiles, according to some embodiments.

FIG. 9 illustrates a diagram of a home with one or more smart-homedevices that capture sound profiles, according to some embodiments.

FIG. 10 illustrates a diagram of smart-home devices in multiple homes incommunication with a management server, according to some embodiments.

FIG. 11 illustrates a chart of how sound profiles can be combined,according to some embodiments.

FIG. 12 illustrates a flowchart of a method for selecting a stored soundprofile for a noise-reduction routine, according to some embodiments.

FIG. 13 illustrates a flow diagram of a converging noise-reductionroutine, according to some embodiments.

FIG. 14 illustrates a timeline of stored sound profiles, according tosome embodiments.

FIG. 15 illustrates a scenario for communication of sound profilesbetween homes, according to some embodiments.

FIG. 16 illustrates a block diagram of an embodiment of a computersystem, according to some embodiments.

FIG. 17 illustrates a block diagram of an embodiment of aspecial-purpose computer, according to some embodiments.

DETAILED DESCRIPTION OF THE INVENTION

The present invention will now be described in detail with reference tocertain embodiments thereof as illustrated in the accompanying drawings.In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known details have not been describedin detail in order not to unnecessarily obscure the present invention.

Provided according to one or more embodiments are methods and systemsfor setting up, pairing, controlling, and/or programming one or more ofintelligent, network-connected, multi-sensing hazard detection units orsmart hazard detectors. These smart hazard detectors may be configuredand adapted to be implemented in a smart home environment, seamlesslyinteracting with other devices in the smart home environment. The term“smart hazard detector” is used herein to represent a particular type ofdevice that can be used for detecting hazards occurring within astructure, e.g., a home, an office or another structure. However, thissmart hazard detector may also be capable of controlling other devices,detecting non-hazard related events (e.g., security related events),and/or working in cooperation with other devices to provide additionalfeatures to the smart home environment. Again, it is within the scope ofthe present teachings for embodiments of the smart hazard detectors ofthe present invention to detect measurable characteristics other thanhazards (e.g., pressure, flow rate, height, position, velocity,acceleration, capacity, power, loudness, and brightness) and monitorand/or respond to one or more measurable characteristics of one or morephysical systems.

It is to be appreciated that “smart home environments” may refer tosmart environments for homes such as a single-family house, but thescope of the present teachings is not so limited, the present teachingsbeing likewise applicable, without limitation, to duplexes, townhomes,multi-unit apartment buildings, hotels, retail stores, office buildings,industrial buildings, and more generally any living space or work spacehaving one or more smart hazard detectors.

It is to be further appreciated that while the terms user, customer,installer, homeowner, occupant, guest, tenant, landlord, repair person,and the like may be used to refer to the person or persons who areinteracting with the smart hazard detector or user interface in thecontext of some particularly advantageous situations described herein,these references are by no means to be considered as limiting the scopeof the present teachings with respect to the person or persons who areperforming such actions. Thus, for example, the terms user, customer,purchaser, installer, subscriber, and homeowner may often refer to thesame person in the case of a single-family residential dwelling, becausethe head of the household is often the person who makes the purchasingdecision, buys the unit, and installs and configures the unit, and isalso one of the users of the unit. However, in other scenarios, such asa landlord-tenant environment, the customer may be the landlord withrespect to purchasing the unit, the installer may be a local apartmentsupervisor, a first user may be the tenant, and a second user may againbe the landlord with respect to remote control functionality.Importantly, while the identity of the person performing the action maybe germane to a particular advantage provided by one or more of theembodiments—for example, the password-protected hazard detectionfunctionality described further herein may be particularly advantageouswhere the landlord holds the sole password and can control hazarddetection via the hazard detection device—such identity should not beconstrued in the descriptions that follow as necessarily limiting thescope of the present teachings to those particular individuals havingthose particular identities.

Overview of Smart Hazard Detector Capabilities

Turning to the figures, FIG. 1 illustrates an example of a smart-homeenvironment 100 within which one or more of the devices, methods,systems, services, and/or computer program products described furtherherein can be applicable. The depicted smart-home environment 100includes a structure 150, which can include, e.g., a house, officebuilding, garage, or mobile home. It will be appreciated that devicescan also be integrated into a smart-home environment 100 that does notinclude an entire structure 150, such as an apartment, condominium, oroffice space. Further, the smart home environment can control and/or becoupled to devices outside of the actual structure 150. Indeed, severaldevices in the smart home environment need not physically be within thestructure 150 at all. For example, a device controlling a pool heater orirrigation system can be located outside of the structure 150.

The depicted structure 150 includes a plurality of rooms 152, separatedat least partly from each other via walls 154. The walls 154 can includeinterior walls or exterior walls. Each room can further include a floor156 and a ceiling 158. Devices can be mounted on, integrated with and/orsupported by a wall 154, floor 156 or ceiling 158.

In some embodiments, the smart-home environment 100 of FIG. 1 includes aplurality of devices, including intelligent, multi-sensing,network-connected devices, that can integrate seamlessly with each otherand/or with a central server or a cloud-computing system to provide anyof a variety of useful smart-home objectives. The smart-home environment100 may include one or more intelligent, multi-sensing,network-connected thermostats 102 (hereinafter referred to as “smartthermostats 102”), one or more intelligent, network-connected,multi-sensing hazard detection units 104 (hereinafter referred to as“smart hazard detectors 104”), and one or more intelligent,multi-sensing, network-connected entryway interface devices 106(hereinafter referred to as “smart doorbells 104”). According toembodiments, the smart thermostat 102 detects ambient climatecharacteristics (e.g., temperature and/or humidity) and controls a HVACsystem 103 accordingly. The smart hazard detector 104 may detect thepresence of a hazardous substance or a substance indicative of ahazardous substance (e.g., smoke, fire, or carbon monoxide). The smartdoorbell 106 may detect a person's approach to or departure from alocation (e.g., an outer door), control doorbell functionality, announcea person's approach or departure via audio or visual means, or controlsettings on a security system (e.g., to activate or deactivate thesecurity system when occupants go and come).

In some embodiments, the smart-home environment 100 of FIG. 1 furtherincludes one or more intelligent, multi-sensing, network-connected wallswitches 108 (hereinafter referred to as “smart wall switches 108”),along with one or more intelligent, multi-sensing, network-connectedwall plug interfaces 110 (hereinafter referred to as “smart wall plugs110”). The smart wall switches 108 may detect ambient lightingconditions, detect room-occupancy states, and control a power and/or dimstate of one or more lights. In some instances, smart wall switches 108may also control a power state or speed of a fan, such as a ceiling fan.The smart wall plugs 110 may detect occupancy of a room or enclosure andcontrol supply of power to one or more wall plugs (e.g., such that poweris not supplied to the plug if nobody is at home).

Still further, in some embodiments, the smart-home environment 100 ofFIG. 1 includes a plurality of intelligent, multi-sensing,network-connected appliances 112 (hereinafter referred to as “smartappliances 112”), such as refrigerators, stoves and/or ovens,televisions, washers, dryers, lights, stereos, intercom systems,garage-door openers, floor fans, ceiling fans, wall air conditioners,pool heaters, irrigation systems, security systems, and so forth.According to embodiments, the network-connected appliances 112 are madecompatible with the smart-home environment by cooperating with therespective manufacturers of the appliances. For example, the appliancescan be space heaters, window AC units, motorized duct vents, etc. Whenplugged in, an appliance can announce itself to the smart-home network,such as by indicating what type of appliance it is, and it canautomatically integrate with the controls of the smart-home. Suchcommunication by the appliance to the smart home can be facilitated byany wired or wireless communication protocols known by those havingordinary skill in the art. The smart home also can include a variety ofnon-communicating legacy appliances 140, such as old conventionalwasher/dryers, refrigerators, and the like which can be controlled,albeit coarsely (ON/OFF), by virtue of the smart wall plugs 110. Thesmart-home environment 100 can further include a variety of partiallycommunicating legacy appliances 142, such as infrared (“IR”) controlledwall air conditioners or other IR-controlled devices, which can becontrolled by IR signals provided by the smart hazard detectors 104 orthe smart wall switches 108.

According to embodiments, the smart thermostats 102, the smart hazarddetectors 104, the smart doorbells 106, the smart wall switches 108, thesmart wall plugs 110, and other devices of the smart-home environment100 are modular and can be incorporated into older and new houses. Forexample, the devices are designed around a modular platform consistingof two basic components: a head unit and a back plate, which is alsoreferred to as a docking station. Multiple configurations of the dockingstation are provided so as to be compatible with any home, such as olderand newer homes. However, all of the docking stations include a standardhead-connection arrangement, such that any head unit can be removablyattached to any docking station. Thus, in some embodiments, the dockingstations are interfaces that serve as physical connections to thestructure and the voltage wiring of the homes, and the interchangeablehead units contain all of the sensors, processors, user interfaces, thebatteries, and other functional components of the devices.

Many different commercial and functional possibilities for provisioning,maintenance, and upgrade are possible. For example, after years of usingany particular head unit, a user will be able to buy a new version ofthe head unit and simply plug it into the old docking station. There arealso many different versions for the head units, such as low-costversions with few features, and then a progression ofincreasingly-capable versions, up to and including extremely fancy headunits with a large number of features. Thus, it should be appreciatedthat the various versions of the head units can all be interchangeable,with any of them working when placed into any docking station. This canadvantageously encourage sharing and re-deployment of old head units—forexample, when an important high-capability head unit, such as a hazarddetector, is replaced by a new version of the head unit, then the oldhead unit can be re-deployed to a backroom or basement, etc. Accordingto embodiments, when first plugged into a docking station, the head unitcan ask the user (by 2D LCD display, 2D/3D holographic projection, voiceinteraction, etc.) a few simple questions such as, “Where am I” and theuser can indicate “living room”, “kitchen” and so forth.

The smart-home environment 100 may also include communication withdevices outside of the physical home but within a proximate geographicalrange of the home. For example, the smart-home environment 100 mayinclude a pool heater monitor 114 that communicates a current pooltemperature to other devices within the smart-home environment 100 orreceives commands for controlling the pool temperature. Similarly, thesmart-home environment 100 may include an irrigation monitor 116 thatcommunicates information regarding irrigation systems within thesmart-home environment 100 and/or receives control information forcontrolling such irrigation systems. According to embodiments, analgorithm is provided for considering the geographic location of thesmart-home environment 100, such as based on the zip code or geographiccoordinates of the home. The geographic information is then used toobtain data helpful for determining optimal times for watering, suchdata may include sun location information, temperature, due point, soiltype of the land on which the home is located, etc.

By virtue of network connectivity, one or more of the smart-home devicesof FIG. 1 can further allow a user to interact with the device even ifthe user is not proximate to the device. For example, a user cancommunicate with a device using a computer (e.g., a desktop computer,laptop computer, or tablet) or other portable electronic device (e.g., asmartphone) 166. A webpage or app can be configured to receivecommunications from the user and control the device based on thecommunications and/or to present information about the device'soperation to the user. For example, the user can view a current setpointtemperature for a device and adjust it using a computer. The user can bein the structure during this remote communication or outside thestructure.

As discussed, users can control the smart thermostat and other smartdevices in the smart-home environment 100 using a network-connectedcomputer or portable electronic device 166. In some examples, some orall of the occupants (e.g., individuals who live in the home) canregister their device 166 with the smart-home environment 100. Suchregistration can be made at a central server to authenticate theoccupant and/or the device as being associated with the home and to givepermission to the occupant to use the device to control the smartdevices in the home. An occupant can use their registered device 166 toremotely control the smart devices of the home, such as when theoccupant is at work or on vacation. The occupant may also use theirregistered device to control the smart devices when the occupant isactually located inside the home, such as when the occupant is sittingon a couch inside the home. It should be appreciated that instead of orin addition to registering devices 166, the smart-home environment 100makes inferences about which individuals live in the home and aretherefore occupants and which devices 166 are associated with thoseindividuals. As such, the smart-home environment “learns” who is anoccupant and permits the devices 166 associated with those individualsto control the smart devices of the home.

In some instances, guests desire to control the smart devices. Forexample, the smart-home environment may receive communication from anunregistered mobile device of an individual inside of the home, wheresaid individual is not recognized as an occupant of the home. Further,for example, a smart-home environment may receive communication from amobile device of an individual who is known to be or who is registeredas a guest.

According to embodiments, a guest-layer of controls can be provided toguests of the smart-home environment 100. The guest-layer of controlsgives guests access to basic controls (e.g., a judicially selectedsubset of features of the smart devices), such as temperatureadjustments, but it locks out other functionalities. The guest layer ofcontrols can be thought of as a “safe sandbox” in which guests havelimited controls, but they do not have access to more advanced controlsthat could fundamentally alter, undermine, damage, or otherwise impairthe occupant-desired operation of the smart devices. For example, theguest layer of controls will not permit the guest to adjust theheat-pump lockout temperature.

A use case example of this is when a guest is in a smart home, the guestcould walk up to the thermostat and turn the dial manually, but theguest may not want to walk around the house “hunting” the thermostat,especially at night while the home is dark and others are sleeping.Further, the guest may not want to go through the hassle of downloadingthe necessary application to their device for remotely controlling thethermostat. In fact, the guest may not have the home owner's logincredentials, etc., and therefore cannot remotely control the thermostatvia such an application. Accordingly, according to embodiments of theinvention, the guest can open a mobile browser on their mobile device,type a keyword, such as “NEST” into the URL field and tap “Go” or“Search”, etc. In response, the device presents the guest with a userinterface which allows the guest to move the target temperature betweena limited range, such as 65 and 80 degrees Fahrenheit. As discussed, theuser interface provides a guest layer of controls that are limited tobasic functions. The guest cannot change the target humidity, modes, orview energy history.

According to embodiments, to enable guests to access the user interfacethat provides the guest layer of controls, a local webserver is providedthat is accessible in the local area network (LAN). It does not requirea password, because physical presence inside the home is establishedreliably enough by the guest's presence on the LAN. In some embodiments,during installation of the smart device, such as the smart thermostat,the home owner is asked if they want to enable a Local Web App (LWA) onthe smart device. Business owners will likely say no; home owners willlikely say yes. When the LWA option is selected, the smart devicebroadcasts to the LAN that the above referenced keyword, such as “NEST”,is now a host alias for its local web server. Thus, no matter whose homea guest goes to, that same keyword (e.g., “NEST”) is always the URL youuse to access the LWA, provided the smart device is purchased from thesame manufacturer. Further, according to embodiments, if there is morethan one smart device on the LAN, the second and subsequent smartdevices do not offer to set up another LWA. Instead, they registerthemselves as target candidates with the master LWA. And in this casethe LWA user would be asked which smart device they want to change thetemperature on before getting the simplified user interface for theparticular smart device they choose.

According to embodiments, a guest layer of controls may also be providedto users by means other than a device 166. For example, the smartdevice, such as the smart thermostat, may be equipped withwalkup-identification technology (e.g., face recognition, RFID,ultrasonic sensors) that “fingerprints” or creates a “signature” for theoccupants of the home. The walkup-identification technology can be thesame as or similar to the fingerprinting and signature creatingtechniques descripted in other sections of this application. Inoperation, when a person who does not live in the home or is otherwisenot registered with the smart home or whose fingerprint or signature isnot recognized by the smart home “walks up” to a smart device, the smartdevice provides the guest with the guest layer of controls, rather thanfull controls.

As described below, the smart thermostat and other smart devices “learn”by observing occupant behavior. For example, the smart thermostat learnsoccupants' preferred temperature set-points for mornings and evenings,and it learns when the occupants are asleep or awake, as well as whenthe occupants are typically away or at home, for example. According toembodiments, when a guest controls the smart devices, such as the smartthermostat, the smart devices do not “learn” from the guest. Thisprevents the guest's adjustments and controls from affecting the learnedpreferences of the occupants.

According to some embodiments, a smart television remote control isprovided. The smart remote control recognizes occupants by thumbprint,visual identification, RFID, etc., and it recognizes a user as a guestor as someone belonging to a particular class having limited control andaccess (e.g., child). Upon recognizing the user as a guest or someonebelonging to a limited class, the smart remote control only permits thatuser to view a subset of channels and to make limited adjustments to thesettings of the television and other devices. For example, a guestcannot adjust the digital video recorder (DVR) settings, and a child islimited to viewing child-appropriate programming.

According to some embodiments, similar controls are provided for otherinstruments, utilities, and devices in the house. For example, sinks,bathtubs, and showers can be controlled by smart spigots that recognizeusers as guests or as children and therefore prevent water fromexceeding a designated temperature that is considered safe.

In some embodiments, in addition to containing processing and sensingcapabilities, each of the devices 102, 104, 106, 108, 110, 112, 114, and116 (collectively referred to as “the smart devices”) is capable of datacommunications and information sharing with any other of the smartdevices, as well as to any central server or cloud-computing system orany other device that is network-connected anywhere in the world. Therequired data communications can be carried out using any of a varietyof custom or standard wireless protocols (Wi-Fi, ZigBee, 6LoWPAN, etc.)and/or any of a variety of custom or standard wired protocols (CAT6Ethernet, HomePlug, etc.)

According to embodiments, all or some of the smart devices can serve aswireless or wired repeaters. For example, a first one of the smartdevices can communicate with a second one of the smart device via awireless router 160. The smart devices can further communicate with eachother via a connection to a network, such as the Internet 162. Throughthe Internet 162, the smart devices can communicate with a centralserver or a cloud-computing system 164. The central server orcloud-computing system 164 can be associated with a manufacturer,support entity, or service provider associated with the device. For oneembodiment, a user may be able to contact customer support using adevice itself rather than needing to use other communication means suchas a telephone or Internet-connected computer. Further, software updatescan be automatically sent from the central server or cloud-computingsystem 164 to devices (e.g., when available, when purchased, or atroutine intervals).

According to embodiments, the smart devices combine to create a meshnetwork of spokesman and low-power nodes in the smart-home environment100, where some of the smart devices are “spokesman” nodes and othersare “low-powered” nodes. Some of the smart devices in the smart-homeenvironment 100 are battery powered, while others have a regular andreliable power source, such as by connecting to wiring (e.g., to 120Vline voltage wires) behind the walls 154 of the smart-home environment.The smart devices that have a regular and reliable power source arereferred to as “spokesman” nodes. These nodes are equipped with thecapability of using any wireless protocol or manner to facilitatebidirectional communication with any of a variety of other devices inthe smart-home environment 100 as well as with the central server orcloud-computing system 164. On the other hand, the devices that arebattery powered are referred to as “low-power” nodes. These nodes tendto be smaller than spokesman nodes and can only communicate usingwireless protocols that requires very little power, such as Zigbee,6LoWPAN, etc. Further, some, but not all, low-power nodes are incapableof bidirectional communication. These low-power nodes send messages, butthey are unable to “listen”. Thus, other devices in the smart-homeenvironment 100, such as the spokesman nodes, cannot send information tothese low-power nodes.

As described, the smart devices serve as low-power and spokesman nodesto create a mesh network in the smart-home environment 100. Individuallow-power nodes in the smart-home environment regularly send outmessages regarding what they are sensing, and the other low-powerednodes in the smart-home environment—in addition to sending out their ownmessages—repeat the messages, thereby causing the messages to travelfrom node to node (i.e., device to device) throughout the smart-homeenvironment 100. The spokesman nodes in the smart-home environment 100are able to “drop down” to low-powered communication protocols toreceive these messages, translate the messages to other communicationprotocols, and send the translated messages to other spokesman nodesand/or the central server or cloud-computing system 164. Thus, thelow-powered nodes using low-power communication protocols are able sendmessages across the entire smart-home environment 100 as well as overthe Internet 162 to the central server or cloud-computing system 164.According to embodiments, the mesh network enables the central server orcloud-computing system 164 regularly receive data from all of the smartdevices in the home, make inferences based on the data, and sendcommands back to one of the smart devices to accomplish some of thesmart-home objectives descried herein.

As described, the spokesman nodes and some of the low-powered nodes arecapable of “listening”. Accordingly, users, other devices, and thecentral server or cloud-computing system 164 can communicate controls tothe low-powered nodes. For example, a user can use the portableelectronic device (e.g., a smartphone) 166 to send commands over theInternet to the central server or cloud-computing system 164, which thenrelays the commands to the spokesman nodes in the smart-home environment100. The spokesman nodes drop down to a low-power protocol tocommunicate the commands to the low-power nodes throughout thesmart-home environment, as well as to other spokesman nodes that did notreceive the commands directly from the central server or cloud-computingsystem 164.

An example of a low-power node is a smart nightlight 170. In addition tohousing a light source, the smart nightlight 170 houses an occupancysensor, such as an ultrasonic or passive IR sensor, and an ambient lightsensor, such as a photoresistor or a single-pixel sensor that measureslight in the room. In some embodiments, the smart nightlight 170 isconfigured to activate the light source when its ambient light sensordetects that the room is dark and when its occupancy sensor detects thatsomeone is in the room. In other embodiments, the smart nightlight 170is simply configured to activate the light source when its ambient lightsensor detects that the room is dark. Further, according to embodiments,the smart nightlight 170 includes a low-power wireless communicationchip (e.g., ZigBee chip) that regularly sends out messages regarding theoccupancy of the room and the amount of light in the room, includinginstantaneous messages coincident with the occupancy sensor detectingthe presence of a person in the room. As mentioned above, these messagesmay be sent wirelessly, using the mesh network, from node to node (i.e.,smart device to smart device) within the smart-home environment 100 aswell as over the Internet 162 to the central server or cloud-computingsystem 164.

Other examples of low-powered nodes include battery-operated versions ofthe smart hazard detectors 104. These smart hazard detectors 104 areoften located in an area without access to constant and reliable powerand, as discussed in detail below, may include any number and type ofsensors, such as smoke/fire/heat sensors, carbon monoxide/dioxidesensors, occupancy/motion sensors, ambient light sensors, temperaturesensors, humidity sensors, and the like. Furthermore, smart hazarddetectors 104 can send messages that correspond to each of therespective sensors to the other devices and the central server orcloud-computing system 164, such as by using the mesh network asdescribed above.

Examples of spokesman nodes include smart doorbells 106, smartthermostats 102, smart wall switches 108, and smart wall plugs 110.These devices 102, 106, 108, and 110 are often located near andconnected to a reliable power source, and therefore can include morepower-consuming components, such as one or more communication chipscapable of bidirectional communication in any variety of protocols.

In some embodiments, these low-powered and spokesman nodes (e.g.,devices 102, 104, 106, 108, 110, 112, and 170) can function as“tripwires” for an alarm system in the smart-home environment. Forexample, in the event a perpetrator circumvents detection by alarmsensors located at windows, doors, and other entry points of thesmart-home environment 100, the alarm could be triggered upon receivingan occupancy, motion, heat, sound, etc. message from one or more of thelow-powered and spokesman nodes in the mesh network. For example, uponreceiving a message from a smart nightlight 170 indicating the presenceof a person, the central server or cloud-computing system 164 or someother device could trigger an alarm, provided the alarm is armed at thetime of detection. Thus, the alarm system could be enhanced by variouslow-powered and spokesman nodes located throughout the smart-homeenvironment 100. In this example, a user could enhance the security ofthe smart-home environment 100 by buying and installing extra smartnightlights 170.

In some embodiments, the mesh network can be used to automatically turnon and off lights as a person transitions from room to room. Forexample, the low-powered and spokesman nodes (e.g., devices 102, 104,106, 108, 110, 112, and 170) detect the person's movement through thesmart-home environment and communicate corresponding messages throughthe mesh network. Using the messages that indicate which rooms areoccupied, the central server or cloud-computing system 164 or some otherdevice activates and deactivates the smart wall switches 108 toautomatically provide light as the person moves from room to room in thesmart-home environment 100. Further, users may provide pre-configurationinformation that indicates which smart wall plugs 110 provide power tolamps and other light sources, such as the smart nightlight 170.Alternatively, this mapping of light sources to wall plugs 110 can bedone automatically (e.g., the smart wall plugs 110 detect when a lightsource is plugged into it, and it sends a corresponding message to thecentral server or cloud-computing system 164). Using this mappinginformation in combination with messages that indicate which rooms areoccupied, the central server or cloud-computing system 164 or some otherdevice activates and deactivates the smart wall plugs 110 that providepower to lamps and other light sources so as to track the person'smovement and provide light as the person moves from room to room.

In some embodiments, the mesh network of low-powered and spokesman nodescan be used to provide exit lighting in the event of an emergency. Insome instances, to facilitate this, users provide pre-configurationinformation that indicates exit routes in the smart-home environment100. For example, for each room in the house, the user provides a map ofthe best exit route. It should be appreciated that instead of a userproviding this information, the central server or cloud-computing system164 or some other device could automatically determine the routes usinguploaded maps, diagrams, architectural drawings of the smart-home house,as well as using a map generated based on positional informationobtained from the nodes of the mesh network (e.g., positionalinformation from the devices is used to construct a map of the house).In operation, when an alarm is activated (e.g., when one or more of thesmart hazard detector 104 detects smoke and activates an alarm), thecentral server or cloud-computing system 164 or some other device usesoccupancy information obtained from the low-powered and spokesman nodesto determine which rooms are occupied and then turns on lights (e.g.,nightlights 170, wall switches 108, wall plugs 110 that power lamps,etc.) along the exit routes from the occupied rooms so as to provideemergency exit lighting.

Further included and illustrated in the exemplary smart-home environment100 of FIG. 1 are service robots 162 each configured to carry out, in anautonomous manner, any of a variety of household tasks. For someembodiments, the service robots 162 can be respectively configured toperform floor sweeping, floor washing, etc. in a manner similar to thatof known commercially available devices such as the ROOMBA™ and SCOOBA™products sold by iRobot, Inc. of Bedford, Mass. Tasks such as floorsweeping and floor washing can be considered as “away” or “while-away”tasks for purposes of the instant description, as it is generally moredesirable for these tasks to be performed when the occupants are notpresent. For other embodiments, one or more of the service robots 162are configured to perform tasks such as playing music for an occupant,serving as a localized thermostat for an occupant, serving as alocalized air monitor/purifier for an occupant, serving as a localizedbaby monitor, serving as a localized hazard detector for an occupant,and so forth, it being generally more desirable for such tasks to becarried out in the immediate presence of the human occupant. Forpurposes of the instant description, such tasks can be considered as“human-facing” or “human-centric” tasks.

When serving as a localized thermostat for an occupant, a particular oneof the service robots 162 can be considered to be facilitating what canbe called a “personal comfort-area network” for the occupant, with theobjective being to keep the occupant's immediate space at a comfortabletemperature wherever that occupant may be located in the home. This canbe contrasted with conventional wall-mounted room thermostats, whichhave the more attenuated objective of keeping a statically-definedstructural space at a comfortable temperature. According to oneembodiment, the localized-thermostat service robot 162 is configured tomove itself into the immediate presence (e.g., within five feet) of aparticular occupant who has settled into a particular location in thehome (e.g. in the dining room to eat their breakfast and read the news).The localized-thermostat service robot 162 includes a temperaturesensor, a processor, and wireless communication components configuredsuch that control communications with the HVAC system, either directlyor through a wall-mounted wirelessly communicating thermostat coupled tothe HVAC system, are maintained and such that the temperature in theimmediate vicinity of the occupant is maintained at their desired level.If the occupant then moves and settles into another location (e.g. tothe living room couch to watch television), the localized-thermostatservice robot 162 proceeds to move and park itself next to the couch andkeep that particular immediate space at a comfortable temperature.

Technologies by which the localized-thermostat service robot 162 (and/orthe larger smart-home system of FIG. 1) can identify and locate theoccupant whose personal-area space is to be kept at a comfortabletemperature can include, but are not limited to, RFID sensing (e.g.,person having an RFID bracelet, RFID necklace, or RFID key fob),synthetic vision techniques (e.g., video cameras and face recognitionprocessors), audio techniques (e.g., voice, sound pattern, vibrationpattern recognition), ultrasound sensing/imaging techniques, andinfrared or near-field communication (NFC) techniques (e.g., personwearing an infrared or NFC-capable smartphone), along with rules-basedinference engines or artificial intelligence techniques that draw usefulconclusions from the sensed information (e.g., if there is only a singleoccupant present in the home, then that is the person whose immediatespace should be kept at a comfortable temperature, and the selection ofthe desired comfortable temperature should correspond to that occupant'sparticular stored profile).

When serving as a localized air monitor/purifier for an occupant, aparticular service robot 162 can be considered to be facilitating whatcan be called a “personal health-area network” for the occupant, withthe objective being to keep the air quality in the occupant's immediatespace at healthy levels. Alternatively or in conjunction therewith,other health-related functions can be provided, such as monitoring thetemperature or heart rate of the occupant (e.g., using finely remotesensors, near-field communication with on-person monitors, etc.). Whenserving as a localized hazard detector for an occupant, a particularservice robot 162 can be considered to be facilitating what can becalled a “personal safety-area network” for the occupant, with theobjective being to ensure there is no excessive carbon monoxide, smoke,fire, etc., in the immediate space of the occupant. Methods analogous tothose described above for personal comfort-area networks in terms ofoccupant identifying and tracking are likewise applicable for personalhealth-area network and personal safety-area network embodiments.

According to some embodiments, the above-referenced facilitation ofpersonal comfort-area networks, personal health-area networks, personalsafety-area networks, and/or other such human-facing functionalities ofthe service robots 162, are further enhanced by logical integration withother smart sensors in the home according to rules-based inferencingtechniques or artificial intelligence techniques for achieving betterperformance of those human-facing functionalities and/or for achievingthose goals in energy-conserving or other resource-conserving ways.Thus, for one embodiment relating to personal health-area networks, theair monitor/purifier service robot 162 can be configured to detectwhether a household pet is moving toward the currently settled locationof the occupant (e.g., using on-board sensors and/or by datacommunications with other smart-home sensors along with rules-basedinferencing/artificial intelligence techniques), and if so, the airpurifying rate is immediately increased in preparation for the arrivalof more airborne pet dander. For another embodiment relating to personalsafety-area networks, the hazard detector service robot 162 can beadvised by other smart-home sensors that the temperature and humiditylevels are rising in the kitchen, which is nearby to the occupant'scurrent dining room location, and responsive to this advisory the hazarddetector service robot 162 will temporarily raise a hazard detectionthreshold, such as a smoke detection threshold, under an inference thatany small increases in ambient smoke levels will most likely be due tocooking activity and not due to a genuinely hazardous condition.

The above-described “human-facing” and “away” functionalities can beprovided, without limitation, by multiple distinct service robots 162having respective dedicated ones of such functionalities, by a singleservice robot 162 having an integration of two or more different ones ofsuch functionalities, and/or any combinations thereof (including theability for a single service robot 162 to have both “away” and “humanfacing” functionalities) without departing from the scope of the presentteachings. Electrical power can be provided by virtue of rechargeablebatteries or other rechargeable methods, with FIG. 1 illustrating anexemplary out-of-the-way docking station 164 to which the service robots162 will automatically dock and recharge its batteries (if needed)during periods of inactivity. Preferably, each service robot 162includes wireless communication components that facilitate datacommunications with one or more of the other wirelessly communicatingsmart-home sensors of FIG. 1 and/or with one or more other servicerobots 162 (e.g., using Wi-Fi, Zigbee, Z-Wave, 6LoWPAN, etc.), and oneor more of the smart-home devices of FIG. 1 can be in communication witha remote server over the Internet. Alternatively or in conjunctiontherewith, each service robot 162 can be configured to communicatedirectly with a remote server by virtue of cellular telephonecommunications, satellite communications, 3G/4G network datacommunications, or other direct communication method.

Provided according to some embodiments are systems and methods relatingto the integration of the service robot(s) 162 with home securitysensors and related functionalities of the smart home system. Theembodiments are particularly applicable and advantageous when appliedfor those service robots 162 that perform “away” functionalities or thatotherwise are desirable to be active when the home is unoccupied(hereinafter “away-service robots”). Included in the embodiments aremethods and systems for ensuring that home security systems, intrusiondetection systems, and/or occupancy-sensitive environmental controlsystems (for example, occupancy-sensitive automated setback thermostatsthat enter into a lower-energy-using condition when the home isunoccupied) are not erroneously triggered by the away-service robots.

Provided according to one embodiment is a home automation and securitysystem (e.g., as shown in FIG. 1) that is remotely monitored by amonitoring service by virtue of automated systems (e.g., cloud-basedservers or other central servers, hereinafter “central server”) that arein data communications with one or more network-connected elements ofthe home automation and security system. The away-service robots areconfigured to be in operative data communication with the centralserver, and are configured such that they remain in a non-away-servicestate (e.g., a dormant state at their docking station) unless permissionis granted from the central server (e.g., by virtue of an“away-service-OK” message from the central server) to commence theiraway-service activities. An away-state determination made by the system,which can be arrived at (i) exclusively by local on-premises smartdevice(s) based on occupancy sensor data, (ii) exclusively by thecentral server based on received occupancy sensor data and/or based onreceived proximity-related information such as GPS coordinates from usersmartphones or automobiles, or (iii) any combination of (i) and (ii) canthen trigger the granting of away-service permission to the away-servicerobots by the central server. During the course of the away-servicerobot activity, during which the away-service robots may continuouslydetect and send their in-home location coordinates to the centralserver, the central server can readily filter signals from the occupancysensing devices to distinguish between the away-service robot activityversus any unexpected intrusion activity, thereby avoiding a falseintrusion alarm condition while also ensuring that the home is secure.Alternatively or in conjunction therewith, the central server mayprovide filtering data (such as an expected occupancy-sensing profiletriggered by the away-service robots) to the occupancy sensing nodes orassociated processing nodes of the smart home, such that the filteringis performed at the local level. Although somewhat less secure, it wouldalso be within the scope of the present teachings for the central serverto temporarily disable the occupancy sensing equipment for the durationof the away-service robot activity.

According to another embodiment, functionality similar to that of thecentral server in the above example can be performed by an on-sitecomputing device such as a dedicated server computer, a “master” homeautomation console or panel, or as an adjunct function of one or more ofthe smart-home devices of FIG. 1. In such an embodiment, there would beno dependency on a remote service provider to provide the“away-service-OK” permission to the away-service robots and thefalse-alarm-avoidance filtering service or filter information for thesensed intrusion detection signals.

According to other embodiments, there are provided methods and systemsfor implementing away-service robot functionality while avoiding falsehome security alarms and false occupancy-sensitive environmentalcontrols without the requirement of a single overall event orchestrator.For purposes of the simplicity in the present disclosure, the homesecurity systems and/or occupancy-sensitive environmental controls thatwould be triggered by the motion, noise, vibrations, or otherdisturbances of the away-service robot activity are referenced simply as“activity sensing systems,” and when so triggered will yield a“disturbance-detected” outcome representative of the false trigger (forexample, an alarm message to a security service, or an “arrival”determination for an automated setback thermostat that causes the hometo be heated or cooled to a more comfortable “occupied” setpointtemperature). According to one embodiment, the away-service robots areconfigured to emit a standard ultrasonic sound throughout the course oftheir away-service activity, the activity sensing systems are configuredto detect that standard ultrasonic sound, and the activity sensingsystems are further configured such that no disturbance-detected outcomewill occur for as long as that standard ultrasonic sound is detected.For other embodiments, the away-service robots are configured to emit astandard notification signal throughout the course of their away-serviceactivity, the activity sensing systems are configured to detect thatstandard notification signal, and the activity sensing systems arefurther configured such that no disturbance-detected outcome will occurfor as long as that standard notification signal is detected, whereinthe standard notification signal comprises one or more of: an opticalnotifying signal; an audible notifying signal; an infrared notifyingsignal; an infrasonic notifying signal; a wirelessly transmitted datanotification signal (e.g., an IP broadcast, multicast, or unicastnotification signal, or a notification message sent in an TCP/IP two-waycommunication session).

According to some embodiments, the notification signals sent by theaway-service robots to the activity sensing systems are authenticatedand encrypted such that the notifications cannot be learned andreplicated by a potential burglar. Any of a variety of knownencryption/authentication schemes can be used to ensure such datasecurity including, but not limited to, methods involving third partydata security services or certificate authorities. For some embodiments,a permission request-response model can be used, wherein any particularaway-service robot requests permission from each activity sensing systemin the home when it is ready to perform its away-service tasks, and doesnot initiate such activity until receiving a “yes” or “permissiongranted” message from each activity sensing system (or from a singleactivity sensing system serving as a “spokesman” for all of the activitysensing systems). One advantage of the described embodiments that do notrequire a central event orchestrator is that there can (optionally) bemore of an arms-length relationship between the supplier(s) of the homesecurity/environmental control equipment, on the one hand, and thesupplier(s) of the away-service robot(s), on the other hand, as it isonly required that there is the described standard one-way notificationprotocol or the described standard two-way request/permission protocolto be agreed upon by the respective suppliers.

According to still other embodiments, the activity sensing systems areconfigured to detect sounds, vibrations, RF emissions, or otherdetectable environmental signals or “signatures” that are intrinsicallyassociated with the away-service activity of each away-service robot,and are further configured such that no disturbance-detected outcomewill occur for as long as that particular detectable signal orenvironmental “signature” is detected. By way of example, a particularkind of vacuum-cleaning away-service robot may emit a specific sound orRF signature. For one embodiment, the away-service environmentalsignatures for each of a plurality of known away-service robots arestored in the memory of the activity sensing systems based onempirically collected data, the environmental signatures being suppliedwith the activity sensing systems and periodically updated by a remoteupdate server. For another embodiment, the activity sensing systems canbe placed into a “training mode” for the particular home in which theyare installed, wherein they “listen” and “learn” the particularenvironmental signatures of the away-service robots for that home duringthat training session, and thereafter will suppress disturbance-detectedoutcomes for intervals in which those environmental signatures areheard.

For still another embodiment, which is particularly useful when theactivity sensing system is associated with occupancy-sensitiveenvironmental control equipment rather than a home security system, theactivity sensing system is configured to automatically learn theenvironmental signatures for the away-service robots by virtue ofautomatically performing correlations over time between detectedenvironmental signatures and detected occupancy activity. By way ofexample, for one embodiment an intelligent automatednonoccupancy-triggered setback thermostat such as the Nest LearningThermostat can be configured to constantly monitor for audible and RFactivity as well as to perform infrared-based occupancy detection. Inparticular view of the fact that the environmental signature of theaway-service robot will remain relatively constant from event to event,and in view of the fact that the away-service events will likely either(a) themselves be triggered by some sort of nonoccupancy condition asmeasured by the away-service robots themselves, or (b) occur at regulartimes of day, there will be patterns in the collected data by which theevents themselves will become apparent and for which the environmentalsignatures can be readily learned. Generally speaking, for thisautomatic-learning embodiment in which the environmental signatures ofthe away-service robots are automatically learned without requiring userinteraction, it is more preferable that a certain number of falsetriggers be tolerable over the course of the learning process.Accordingly, this automatic-learning embodiment is more preferable forapplication in occupancy-sensitive environmental control equipment (suchas an automated setback thermostat) rather than home security systemsfor the reason that a few false occupancy determinations may cause a fewinstances of unnecessary heating or cooling, but will not otherwise haveany serious consequences, whereas false home security alarms may havemore serious consequences.

According to embodiments, technologies including the sensors of thesmart devices located in the mesh network of the smart-home environmentin combination with rules-based inference engines or artificialintelligence provided at the central server or cloud-computing system164 are used to provide a personal “smart alarm clock” for individualoccupants of the home. For example, user-occupants can communicate withthe central server or cloud-computing system 164 via their mobiledevices 166 to access an interface for the smart alarm clock. There,occupants can turn on their “smart alarm clock” and input a wake timefor the next day and/or for additional days. In some embodiments, theoccupant may have the option of setting a specific wake time for eachday of the week, as well as the option of setting some or all of theinputted wake times to “repeat”. Artificial intelligence will be used toconsider the occupant's response to these alarms when they go off andmake inferences about the user's preferred sleep patterns over time.

According to embodiments, the smart device in the smart-home environment100 that happens to be closest to the occupant when the occupant fallsasleep will be the device that transmits messages regarding when theoccupant stopped moving, from which the central server orcloud-computing system 164 will make inferences about where and when theoccupant prefers to sleep. This closest smart device will as be thedevice that sounds the alarm to wake the occupant. In this manner, the“smart alarm clock” will follow the occupant throughout the house, bytracking the individual occupants based on their “unique signature”,which is determined based on data obtained from sensors located in thesmart devices. For example, the sensors include ultrasonic sensors,passive IR sensors, and the like. The unique signature is based on acombination of walking gate, patterns of movement, voice, height, size,etc. It should be appreciated that facial recognition may also be used.

According to an embodiment, the wake times associated with the “smartalarm clock” are used by the smart thermostat 102 to control the HVAC inan efficient manner so as to pre-heat or cool the house to theoccupant's desired “sleeping” and “awake” temperature settings. Thepreferred settings can be learned over time, such as by observing whichtemperature the occupant sets the thermostat to before going to sleepand which temperature the occupant sets the thermostat to upon wakingup.

According to an embodiment, a device is positioned proximate to theoccupant's bed, such as on an adjacent nightstand, and collects data asthe occupant sleeps using noise sensors, motion sensors (e.g.,ultrasonic, IR, and optical), etc. Data may be obtained by the othersmart devices in the room as well. Such data may include the occupant'sbreathing patterns, heart rate, movement, etc. Inferences are made basedon this data in combination with data that indicates when the occupantactually wakes up. For example, if—on a regular basis—the occupant'sheart rate, breathing, and moving all increase by 5% to 10%, twenty tothirty minutes before the occupant wakes up each morning, thenpredictions can be made regarding when the occupant is going to wake.Other devices in the home can use these predictions to provide othersmart-home objectives, such as adjusting the smart thermostat 102 so asto pre-heat or cool the home to the occupant's desired setting beforethe occupant wakes up. Further, these predictions can be used to set the“smart alarm clock” for the occupant, to turn on lights, etc.

According to embodiments, technologies including the sensors of thesmart devices located throughout the smart-home environment incombination with rules-based inference engines or artificialintelligence provided at the central server or cloud-computing system164 are used to detect or monitor the progress of Alzheimer's Disease.For example, the unique signatures of the occupants are used to trackthe individual occupants' movement throughout the smart-home environment100. This data can be aggregated and analyzed to identify patternsindicative of Alzheimer's. Oftentimes, individuals with Alzheimer's havedistinctive patterns of migration in their homes. For example, a personwill walk to the kitchen and stand there for a while, then to the livingroom and stand there for a while, and then back to the kitchen. Thispattern will take about thirty minutes, and then the person will repeatthe pattern. According to embodiments, the remote servers or cloudcomputing architectures 164 analyze the person's migration datacollected by the mesh network of the smart-home environment to identifysuch patterns.

FIG. 2 illustrates a network-level view of an extensible devices andservices platform 200 with which a plurality of smart-home environments,such as the smart-home environment 100 of FIG. 1, can be integrated. Theextensible devices and services platform 200 includes remote servers orcloud computing architectures 164. Each of the intelligent,network-connected devices 102, 104, 106, 108, 110, 112, 114, and 116from FIG. 1 (identified simply as “smart devices” in FIGS. 2-3 herein)can communicate with the remote servers or cloud computing architectures164. For example, a connection to the Internet 162 can be establishedeither directly (for example, using 3G/4G connectivity to a wirelesscarrier), through a hubbed network 212 (which can be a scheme rangingfrom a simple wireless router, for example, up to and including anintelligent, dedicated whole-home control node), or through anycombination thereof.

Although in some examples provided herein, the devices and servicesplatform 200 communicates with and collects data from the smart devicesof smart-home environment 100 of FIG. 1, it should be appreciated thatthe devices and services platform 200 communicates with and collectsdata from a plurality of smart-home environments across the world. Forexample, the central server or cloud-computing system 164 can collecthome data 202 from the devices of one or more smart-home environments,where the devices can routinely transmit home data or can transmit homedata in specific instances (e.g., when a device queries the home data202). Thus, the devices and services platform 200 routinely collectsdata from homes across the world. As described, the collected home data202 includes, for example, power consumption data, occupancy data, HVACsettings and usage data, carbon monoxide levels data, carbon dioxidelevels data, volatile organic compounds levels data, sleeping scheduledata, cooking schedule data, inside and outside temperature humiditydata, television viewership data, inside and outside noise level data,etc.

The central server or cloud-computing architecture 164 can furtherprovide one or more services 204. The services 204 can include, e.g.,software updates, customer support, sensor data collection/logging,remote access, remote or distributed control, or use suggestions (e.g.,based on collected home data 202 to improve performance, reduce utilitycost, etc.). Data associated with the services 204 can be stored at thecentral server or cloud-computing system 164 and the central server orthe cloud-computing system 164 can retrieve and transmit the data at anappropriate time (e.g., at regular intervals, upon receiving a requestfrom a user, etc.).

As illustrated in FIG. 2, an embodiment of the extensible devices andservices platform 200 includes a processing engine 206, which can beconcentrated at a single server or distributed among several differentcomputing entities without limitation. The processing engine 206 caninclude engines configured to receive data from devices of smart-homeenvironments (e.g., via the Internet or a hubbed network), to index thedata, to analyze the data and/or to generate statistics based on theanalysis or as part of the analysis. The analyzed data can be stored asderived home data 208.

Results of the analysis or statistics can thereafter be transmitted backto the device that provided home data used to derive the results, toother devices, to a server providing a webpage to a user of the device,or to other non-device entities. For example, use statistics, usestatistics relative to use of other devices, use patterns, and/orstatistics summarizing sensor readings can be generated by theprocessing engine 206 and transmitted. The results or statistics can beprovided via the Internet 162. In this manner, the processing engine 206can be configured and programmed to derive a variety of usefulinformation from the home data 202. A single server can include one ormore engines.

The derived data can be highly beneficial at a variety of differentgranularities for a variety of useful purposes, ranging from explicitprogrammed control of the devices on a per-home, per-neighborhood, orper-region basis (for example, demand-response programs for electricalutilities), to the generation of inferential abstractions that canassist on a per-home basis (for example, an inference can be drawn thatthe homeowner has left for vacation and so security detection equipmentcan be put on heightened sensitivity), to the generation of statisticsand associated inferential abstractions that can be used for governmentor charitable purposes. For example, processing engine 206 can generatestatistics about device usage across a population of devices and sendthe statistics to device users, service providers or other entities(e.g., that have requested or may have provided monetary compensationfor the statistics).

According to some embodiments, the home data 202, the derived home data208, and/or another data can be used to create “automated neighborhoodsafety networks.” For example, in the event the central server orcloud-computing architecture 164 receives data indicating that aparticular home has been broken into, is experiencing a fire, or someother type of emergency event, an alarm is sent to other smart homes inthe “neighborhood.” In some instances, the central server orcloud-computing architecture 164 automatically identifies smart homeswithin a radius of the home experiencing the emergency and sends analarm to the identified homes. In such instances, the other homes in the“neighborhood” do not have to sign up for or register to be a part of asafety network, but instead are notified of an emergency based on theirproximity to the location of the emergency. This creates robust andevolving neighborhood security watch networks, such that if one person'shome is getting broken into, an alarm can be sent to nearby homes, suchas by audio announcements via the smart devices located in those homes.It should be appreciated that this can be an opt-in service and that, inaddition to or instead of the central server or cloud-computingarchitecture 164 selecting which homes to send alerts to, individualscan subscribe to participate in such networks and individuals canspecify which homes they want to receive alerts from. This can include,for example, the homes of family members who live in different cities,such that individuals can receive alerts when their loved ones in otherlocations are experiencing an emergency.

According to some embodiments, sound, vibration, and/or motion sensingcomponents of the smart devices are used to detect sound, vibration,and/or motion created by running water. Based on the detected sound,vibration, and/or motion, the central server or cloud-computingarchitecture 164 makes inferences about water usage in the home andprovides related services. For example, the central server orcloud-computing architecture 164 can run programs/algorithms thatrecognize what water sounds like and when it is running in the home.According to one embodiment, to map the various water sources of thehome, upon detecting running water, the central server orcloud-computing architecture 164 sends a message an occupant's mobiledevice asking if water is currently running or if water has beenrecently run in the home and, if so, which room and whichwater-consumption appliance (e.g., sink, shower, toilet, etc.) was thesource of the water. This enables the central server or cloud-computingarchitecture 164 to determine the “signature” or “fingerprint” of eachwater source in the home. This is sometimes referred to herein as “audiofingerprinting water usage.”

In one illustrative example, the central server or cloud-computingarchitecture 164 creates a signature for the toilet in the masterbathroom, and whenever that toilet is flushed, the central server orcloud-computing architecture 164 will know that the water usage at thattime is associated with that toilet. Thus, the central server orcloud-computing architecture 164 can track the water usage of thattoilet as well as each water-consumption application in the home. Thisinformation can be correlated to water bills or smart water meters so asto provide users with a breakdown of their water usage.

According to some embodiments, sound, vibration, and/or motion sensingcomponents of the smart devices are used to detect sound, vibration,and/or motion created by mice and other rodents as well as by termites,cockroaches, and other insects (collectively referred to as “pests”).Based on the detected sound, vibration, and/or motion, the centralserver or cloud-computing architecture 164 makes inferences aboutpest-detection in the home and provides related services. For example,the central server or cloud-computing architecture 164 can runprograms/algorithms that recognize what certain pests sound like, howthey move, and/or the vibration they create, individually and/orcollectively. According to one embodiment, the central server orcloud-computing architecture 164 can determine the “signatures” ofparticular types of pests.

For example, in the event the central server or cloud-computingarchitecture 164 detects sounds that may be associated with pests, itnotifies the occupants of such sounds and suggests hiring a pest controlcompany. If it is confirmed that pests are indeed present, the occupantsinput to the central server or cloud-computing architecture 164 confirmsthat its detection was correct, along with details regarding theidentified pests, such as name, type, description, location, quantity,etc. This enables the central server or cloud-computing architecture 164to “tune” itself for better detection and create “signatures” or“fingerprints” for specific types of pests. For example, the centralserver or cloud-computing architecture 164 can use the tuning as well asthe signatures and fingerprints to detect pests in other homes, such asnearby homes that may be experiencing problems with the same pests.Further, for example, in the event that two or more homes in a“neighborhood” are experiencing problems with the same or similar typesof pests, the central server or cloud-computing architecture 164 canmake inferences that nearby homes may also have such problems or may besusceptible to having such problems, and it can send warning messages tothose homes to help facilitate early detection and prevention.

In some embodiments, to encourage innovation and research and toincrease products and services available to users, the devices andservices platform 200 expose a range of application programminginterfaces (APIs) 210 to third parties, such as charities 222,governmental entities 224 (e.g., the Food and Drug Administration or theEnvironmental Protection Agency), academic institutions 226 (e.g.,university researchers), businesses 228 (e.g., providing devicewarranties or service to related equipment, targeting advertisementsbased on home data), utility companies 230, and other third parties. TheAPIs 210 are coupled to and permit third-party systems to communicatewith the central server or the cloud-computing system 164, including theservices 204, the processing engine 206, the home data 202, and thederived home data 208. For example, the APIs 210 allow applicationsexecuted by the third parties to initiate specific data processing tasksthat are executed by the central server or the cloud-computing system164, as well as to receive dynamic updates to the home data 202 and thederived home data 208.

For example, third parties can develop programs and/or applications,such as web or mobile apps, that integrate with the central server orthe cloud-computing system 164 to provide services and information tousers. Such programs and application may be, for example, designed tohelp users reduce energy consumption, to preemptively service faultyequipment, to prepare for high service demands, to track past serviceperformance, etc., or to perform any of a variety of beneficialfunctions or tasks now known or hereinafter developed.

According to some embodiments, third-party applications make inferencesfrom the home data 202 and the derived home data 208, such inferencesmay include when are occupants home, when are they sleeping, when arethey cooking, when are they in the den watching television, and when dothey shower. The answers to these questions may help third-partiesbenefit consumers by providing them with interesting information,products and services as well as with providing them with targetedadvertisements.

In one example, a shipping company creates an application that makesinferences regarding when people are at home. The application uses theinferences to schedule deliveries for times when people will most likelybe at home. The application can also build delivery routes around thesescheduled times. This reduces the number of instances where the shippingcompany has to make multiple attempts to deliver packages, and itreduces the number of times consumers have to pick up their packagesfrom the shipping company.

FIG. 3 illustrates an abstracted functional view of the extensibledevices and services platform 200 of FIG. 2, with particular referenceto the processing engine 206 as well as devices, such as those of thesmart-home environment 100 of FIG. 1. Even though devices situated insmart-home environments will have an endless variety of differentindividual capabilities and limitations, they can all be thought of assharing common characteristics in that each of them is a data consumer302 (DC), a data source 304 (DS), a services consumer 306 (SC), and aservices source 308 (SS). Advantageously, in addition to providing theessential control information needed for the devices to achieve theirlocal and immediate objectives, the extensible devices and servicesplatform 200 can also be configured to harness the large amount of datathat is flowing out of these devices. In addition to enhancing oroptimizing the actual operation of the devices themselves with respectto their immediate functions, the extensible devices and servicesplatform 200 can be directed to “repurposing” that data in a variety ofautomated, extensible, flexible, and/or scalable ways to achieve avariety of useful objectives. These objectives may be predefined oradaptively identified based on, e.g., usage patterns, device efficiency,and/or user input (e.g., requesting specific functionality).

For example, FIG. 3 shows processing engine 206 as including a number ofparadigms 310. Processing engine 206 can include a managed servicesparadigm 310 a that monitors and manages primary or secondary devicefunctions. The device functions can include ensuring proper operation ofa device given user inputs, estimating that (e.g., and responding to aninstance in which) an intruder is or is attempting to be in a dwelling,detecting a failure of equipment coupled to the device (e.g., a lightbulb having burned out), implementing or otherwise responding to energydemand response events, or alerting a user of a current or predictedfuture event or characteristic. Processing engine 206 can furtherinclude an advertising/communication paradigm 310 b that estimatescharacteristics (e.g., demographic information), desires and/or productsof interest of a user based on device usage. Services, promotions,products or upgrades can then be offered or automatically provided tothe user. Processing engine 206 can further include a social paradigm310 c that uses information from a social network, provides informationto a social network (for example, based on device usage), and/orprocesses data associated with user and/or device interactions with thesocial network platform. For example, a user's status as reported totheir trusted contacts on the social network could be updated toindicate when they are home based on light detection, security systeminactivation or device usage detectors. As another example, a user maybe able to share device-usage statistics with other users. In yetanother example, a user may share HVAC settings that result in low powerbills and other users may download the HVAC settings to their smartthermostat 102 to reduce their power bills.

The processing engine 206 can include achallenges/rules/compliance/rewards paradigm 310 d that informs a userof challenges, competitions, rules, compliance regulations and/orrewards and/or that uses operation data to determine whether a challengehas been met, a rule or regulation has been complied with and/or areward has been earned. The challenges, rules or regulations can relateto efforts to conserve energy, to live safely (e.g., reducing exposureto toxins or carcinogens), to conserve money and/or equipment life, toimprove health, etc. For example, one challenge may involve participantsturning down their thermostat by one degree for one week. Those thatsuccessfully complete the challenge are rewarded, such as by coupons,virtual currency, status, etc. Regarding compliance, an example involvesa rental-property owner making a rule that no renters are permitted toaccess certain owner's rooms. The devices in the room having occupancysensors could send updates to the owner when the room is accessed.

The processing engine 206 can integrate or otherwise utilize extrinsicinformation 316 from extrinsic sources to improve the functioning of oneor more processing paradigms. Extrinsic information 316 can be used tointerpret data received from a device, to determine a characteristic ofthe environment near the device (e.g., outside a structure that thedevice is enclosed in), to determine services or products available tothe user, to identify a social network or social-network information, todetermine contact information of entities (e.g., public-service entitiessuch as an emergency-response team, the police or a hospital) near thedevice, etc., to identify statistical or environmental conditions,trends or other information associated with a home or neighborhood, andso forth.

An extraordinary range and variety of benefits can be brought about by,and fit within the scope of, the described extensible devices andservices platform 200, ranging from the ordinary to the profound. Thus,in one “ordinary” example, each bedroom of the smart-home environment100 can be provided with a smart wall switch 108, a smart wall plug 110,and/or smart hazard detectors 104, all or some of which include anoccupancy sensor, wherein the occupancy sensor is also capable ofinferring (e.g., by virtue of motion detection, facial recognition,audible sound patterns, etc.) whether the occupant is asleep or awake.If a serious fire event is sensed, the remote security/monitoringservice or fire department is advised of how many occupants there are ineach bedroom, and whether those occupants are still asleep (or immobile)or whether they have properly evacuated the bedroom. While this is, ofcourse, a very advantageous capability accommodated by the describedextensible devices and services platform, there can be substantiallymore “profound” examples that can truly illustrate the potential of alarger “intelligence” that can be made available. By way of perhaps amore “profound” example, the same bedroom occupancy data that is beingused for fire safety can also be “repurposed” by the processing engine206 in the context of a social paradigm of neighborhood childdevelopment and education. Thus, for example, the same bedroom occupancyand motion data discussed in the “ordinary” example can be collected andmade available (properly anonymized) for processing in which the sleeppatterns of schoolchildren in a particular ZIP code can be identifiedand tracked. Localized variations in the sleeping patterns of theschoolchildren may be identified and correlated, for example, todifferent nutrition programs in local schools.

Referring now to FIG. 4A and FIG. 4B, illustrated is a hazard detector400 that may be used as part of a smart home environment 100 aspreviously described. FIG. 4A illustrates an exploded perspective viewof the hazard detector 400, while FIG. 4B illustrates an assembled viewof the same hazard detector 400. In one embodiment, hazard detector 400is a smoke detector that is configured to detect the presence of smokeand sound an alarm to audibly warn an occupant or occupants of the homeor structure of a potential fire or other danger. In other embodiments,hazard detector 400 may be a carbon monoxide detector, heat detector,and the like. In one embodiment, hazard detector 400 is a multi-sensingdetector that includes a smoke detector, carbon monoxide detector, heatdetector, motion detector, and the like. Many of the present teachingsare particularly advantageous for embodiments in which the hazarddetector 400 is a multi-sensing detector, particularly since combiningthe various sensing modes together into a single device can posesubstantial challenges with respect to one or more of devicecompactness, component powering, and overall component governance andcoordination.

For convenience in describing the embodiments herein, the device 400will be referred to hereinbelow as smart hazard detector or hazarddetector 400, although it should be realized that hazard detector 400may include various other devices and that the scope of the presentteachings is not necessarily limited to hazard detectors in which smokeis required as one of the anomalies to be detected. Thus, for example,depending on the particular context as would be apparent to a personskilled in the art upon reading the instant disclosure, one or more ofthe advantageous features and embodiments described herein may bereadily applicable to a multifunctional hazard sensor that detectscarbon monoxide and motion only, or pollen and motion only, or noisepollution and pollen only, and so forth. Nevertheless, the combining ofsmoke detection functionality with other sensing functions does bringabout one or more particularly problematic issues that are addressed byone or more of the present teachings.

In one embodiment, hazard detector 400 is a roughly square orrectangular shaped object having a width of approximately 120 to 134 mmand a thickness of approximately 38 mm. Stated differently, hazarddetector 400 is a multi-sensing unit having a fairly compact shape andsize that may be easily attached to a wall or ceiling of a home orstructure so as to be able, among other functionalities, to detect thepresence of smoke and alert an occupant therein of the potential firedanger. As shown in FIG. 4A, hazard detector 400 includes a mountingplate 410 that may be attached to a wall of the building or structure tosecure the hazard detector 400 thereto. Hazard detector 400 alsoincludes a back plate 420 that may be mounted to the mounting plate 410and a front casing 460 that may be coupled with or otherwise secured toback plate 420 to define a housing having an interior region withinwhich components of the hazard detector 400 are contained. A circuitboard 500 may be coupled with or attached to back plate 420. Variouscomponents may be mounted on circuit board 500. For example, a smokechamber 430 may be coupled with or mounted on circuit board 500 andconfigured to detect the presence of smoke. In one embodiment, smokechamber 430 may be mid-mounted relative to circuit board 500 so that airmay flow into smoke chamber 430 from a position above circuit board 500and below circuit board 500. A speaker 550 and alarm device (notnumbered) may also be mounted on circuit board 500 to audibly warn anoccupant of a potential fire danger when the presence of smoke isdetected via smoke chamber 430. Other components, such as a motionsensor, carbon monoxide sensor, microprocessor, and the like maylikewise be mounted on circuit board 500 as described herein.

In one embodiment, a protective plate 440 may be attached to orotherwise coupled with circuit board 500 to provide a visually pleasingappearance to the inner components of hazard detector 400 and/or tofunnel or direct airflow to smoke chamber 430. For example, when a userviews the internal components of hazard detector 400, such as throughvents in back plate 420, protective plate 440 may provide the appearanceof a relatively smooth surface and otherwise hide the components orcircuitry of circuit board 500. Protective plate 440 may likewisefunction to direct a flow of air from the vents of back plate 420 towardsmoke chamber 430 so as to facilitate air flow into and out of smokechamber 430.

Hazard detector 400 may also include a battery pack 450 that isconfigured to provide power to the various components of hazard detector400 when hazard detector 400 is not coupled with an external powersource, such as a 120 V power source of the home or structure. In someembodiments, a cover plate 470 may be coupled with the front casing 460to provide a visually pleasing appearance to hazard detector 400 and/orfor other functional purposes. In a specific embodiment, cover plate 470may include a plurality of holes or openings that allow one or moresensors coupled with circuit board 500 to view or see through a surfaceof cover plate 470 so as to sense objects external to hazard detector400. The plurality of openings of cover plate 470 may be arranged toprovide a visually pleasing appearance when viewed by occupants of thehome or structure. In one embodiment, the plurality of openings of coverplate 470 may be arranged according to a repeating pattern, such as aFibonacci or other sequence.

A lens button 600 may be coupled with or otherwise mounted to coverplate 470. Lens button 600 may allow one or more sensors to view throughthe lens button 600 for various purposes. For example, in one embodimenta passive IR sensor (not shown) may be positioned behind the lens button600 and configured to view through the lens button 600 to detect thepresence of an occupant or occupants within the home or structure. Insome embodiments, lens button 600 may also function as a button that ispressable by a user to input various commands to hazard detector 400,such as to shut off an alarm that is triggered in response to a false orotherwise harmless condition. Positioned distally behind lens button 600may be a light ring 620 that is configured to receive light, such asfrom an LED or another light emitting element, and disperse the lightwithin ring 620 to provide a desired visual appearance, such as a halobehind lens button 600. Positioned distally behind light ring 620 may bea flexible circuit board 640 that includes one or more electricalcomponents, such as a passive IR sensor (hereinafter PIR sensor), LEDs,and the like. Flexible circuit board 640 (hereinafter flex ring 640) maybe electrically coupled with circuit board 500 to communicate and/orreceive instructions from one or more microprocessors mounted on acircuit board (not shown) during operation of hazard detector 400.Additional details of the components of hazard detector 400 aredescribed in FIGS. 5A-D and 6A-F.

FIG. 4B illustrates hazard detector 400 with the various componentsassembled. Specifically, this figure shows the mounting plate 410, frontcasing 460, back plate 420, and cover plate 470 in an assembledconfiguration with the various other components contained within aninterior space of hazard detector 400. This figure also shows theplurality of holes or openings of cover plate 470 forming a visuallypleasing design that is viewable by occupant of a room within which thehazard detector 400 is mounted. The lens button 600 is shown attached tothe hazard detector 400 so as to be centrally positioned with respect tocover plate 470. As briefly described, light ring 620 may be used toprovide a halo appearance of light around and behind lens button 600.The assembled hazard detector 400 provides a compact yet multifunctionaldevice.

Referring now to FIG. 5A and FIG. 5B, illustrated are front and rearperspective views of circuit board 500. Circuit board 500 includes amain body 502 having a front side or surface and a rear side or surface.As described herein, various electrical components are mounted oncircuit board 500. In some embodiments, these components may be mountedon the front surface of circuit board 500, on the rear surface ofcircuit board 500 opposite the front surface, or on both surfaces of thecircuit board 500. For example, in a specific embodiment one or moremicroprocessors and/or other processor related components may be mountedon the rear surface of circuit board 500 facing protective plate 440while one or more functional components (e.g. an alarm device, COdetector, speaker, motion sensors, Wi-Fi device, Zigbee device, and thelike) are mounted on a front surface of circuit board 500 facing a roomof the home or structure in which the hazard detector 400 is positioned.Other components may be mid-mounted relative to circuit board 500 sothat opposing surfaces are positioned on opposing sides of the circuitboard 500 as described herein.

As shown in FIG. 5A, in a specific embodiment the front surface ofcircuit board 500 may include a CO detector 570 that is configured todetect the presence of carbon monoxide gas and trigger an alarm device560 if the carbon monoxide gas levels are determined to be too high. Thealarm device 560 (which can be a piezoelectric buzzer having anintentionally shrill or jarring sound) may likewise be mounted on thefront surface of circuit board 500 so as to face an occupant of the roomin which the hazard detector 400 is positioned to alarm the occupant ofa potential danger. Alarm device 560 may be configured to produce one ormore sounds or signals to alert the occupant of the potential danger.The front surface may further include an area 552 in which a speaker 550is positioned. Speaker 550 may be configured to provide audible warningsor messages to the occupant of the room. For example, speaker 550 mayalert the occupant of a potential danger and instruct the occupant toexit the room. In some embodiments, speaker 550 may provide specificinstructions to the occupant, such as an exit route to use when exitingthe room and/or home or structure. Other messages may likewise becommunicated to the occupant, such as to alert the occupant that thebatteries are low, that CO levels are relatively high in the room, thathazard detector 400 needs periodic cleaning, or alert the occupant ofany other abnormalities or issues related to hazard detector 400 orcomponents thereof.

Circuit board 500 may also include one or more motion sensors mounted onthe front surface thereof. The motion sensors may be used to determinethe presence of an individual within a room or surrounding area ofhazard detector 400. This information may be used to change thefunctionality of hazard detector 400 and/or one or more other devicesconnected in a common network as described previously. For example, thisinformation may be relayed to a smart thermostat to inform thethermostat that occupants of the home or structure are present so thatthe smart thermostat may condition the home or structure according toone or more learned or programmed settings. Hazard detector 400 maylikewise use this information for one or more purposes, such as to quietthe alarm device (e.g. gesture hush) as described herein or for variousother reasons.

In one embodiment, a first ultrasonic sensor 572 and a second ultrasonicsensor 574 may be mounted on the front surface of circuit board 500. Thetwo ultrasonic sensors, 572 and 574, may be offset axially so as topoint in slightly different directions. In this orientation, eachultrasonic sensor may be used to detect the motion of an individualbased on an orientation of the hazard detector 400 relative to the roomand/or occupant. Detecting the motion of the individual may be used toquiet the alarm device as described herein (i.e., gesture hush) or forany other reason. In one embodiment, an axis of the first ultrasonicsensor 572 may be oriented substantially outward relative to hazarddetector 400 while an axis of the second ultrasonic sensor 574 isoriented at an angle relative to the axis of first ultrasonic sensor572. The first ultrasonic sensor 572 may sense motion of an individualwhen the hazard detector 400 is mounted on a ceiling of the home orstructure. Because the first ultrasonic sensor 572 is orientedsubstantially outward relative to hazard detector 400, the firstultrasonic sensor 572 essentially looks straight down on individualsbeneath hazard detector 400. The second ultrasonic sensor 574 maysimilarly sense motion of the individual when the hazard detector 400 ismounted on a wall of the home or structure. Because the secondultrasonic sensor 574 is oriented at an angle relative to the firstultrasonic sensor 572 and hazard detector 400, the second ultrasonicsensor essentially looks downward toward the floor when the hazarddetector 400 is mounted on a wall of the home or structure, rather thanlooking directly outward as first ultrasonic sensor 572. In oneembodiment, the angular offset of the two ultrasonic sensors may beapproximately 30° or any other desired value.

In another embodiment, the two ultrasonic sensors, 572 and 574, may bereplaced by a single ultrasonic sensor that is configured to rotatewithin hazard detector 400 so that the single ultrasonic sensor iscapable of looking straight outward similar to first ultrasonic sensor572 or capable of looking downward similar to second ultrasonic sensor574. The single ultrasonic sensor may be coupled to circuit board 500via a hinge that allows the ultrasonic sensor to rotate based on theorientation of hazard detector 400. For example, when hazard detector400 is mounted to a ceiling of the home or structure, gravity may orientthe ultrasonic sensor so as to look straight downward; whereas whenhazard detector 400 is coupled to a wall of the home or structure,gravity may cause the ultrasonic sensor to rotate via the hinge and lookdownward toward a floor and relative to hazard detector 400. In anotherembodiment, a motor may be coupled with the single ultrasonic sensor soas to rotate the ultrasonic sensor based on the orientation of hazarddetector 400. In this manner, the ultrasonic sensor may always point ina direction that is likely to detect motion of an individual within theroom or space surrounding the hazard detector 400. In yet anotherembodiment, the single ultrasonic sensor may have a wide field of viewthat is able to substantially accommodate both mounting positions of thetwo ultrasonic sensors 572 and 574.

As shown in FIGS. 5A and 5B, body 502 of circuit board 500 also includesa substantially centrally located aperture 504 through which smokechamber 430 is inserted so as to mid-mount the smoke chamber 430relative to circuit board 500. Aperture 504 may also include a pair ofnotches 506 through which wires are inserted to electrically couple thesmoke chamber 430 with circuit board 500. As previously described,mid-mounting of the smoke chamber 430 through an aperture 504 allowssmoke and air to enter smoke chamber 430 from both the front surface orside of circuit board 500 and the rear surface or side of circuit board500. Various aspects of the electrical components on the circuit board500 are now described, the positions thereon of many of which will beapparent to the skilled reader in view of the descriptions herein andFIGS. 5A-5B. Included on the circuit board 500 can be severalcomponents, including a system processor, relatively high-power wirelesscommunications circuitry and antenna, relatively low-power wirelesscommunications circuitry and antenna, non-volatile memory, audio speaker550, one or more interface sensors, a safety processor, safety sensors,alarm device 560, a power source, and powering circuitry. The componentsare operative to provide failsafe safety detection features and userinterface features using circuit topology and power budgeting methodsthat minimize power consumption. According to one preferred embodiment,a bifurcated or hybrid processor circuit topology is used for handlingthe various features of the hazard detector 400, wherein the safetyprocessor is a relatively small, relatively lean processor that isdedicated to core safety sensor governance and core alarmingfunctionality as would be provided on a conventional smoke/CO alarm, andwherein the system processor is a relatively larger, relativelyhigher-powered processor that is dedicated to more advanced featuressuch as cloud communications, user interface features, occupancy andother advanced environmental tracking features, and more generally anyother task that would not be considered a “core” or “conventional”safety sensing and alarming task.

By way of example and not by way of limitation, the safety processor maybe a Freescale KL15 microcontroller, while the system processor may be aFreescale K60 microcontroller. Preferably, the safety processor isprogrammed and configured such that it is capable of operating andperforming its core safety-related duties regardless of the status orstate of the system processor. Thus, for example, even if the systemprocessor is not available or is otherwise incapable of performing anyfunctions, the safety processor will continue to perform its coresafety-related tasks such that the hazard detector 400 still meets allindustry and/or government safety standards that are required for thesmoke, CO, and/or other safety-related monitoring for which the hazarddetector 400 is offered (provided, of course, that there is sufficientelectrical power available for the safety processor to operate). Thesystem processor, on the other hand, performs what might be called“optional” or “advanced” functions that are overlaid onto thefunctionality of the safety processor, where “optional” or “advanced”refers to tasks that are not specifically required for compliance withindustry and/or governmental safety standards. Thus, although the systemprocessor is designed to interoperate with the safety processor in amanner that can improve the overall performance, feature set, and/orfunctionality of the hazard detector 400, its operation is not requiredin order for the hazard detector 400 to meet core safety-relatedindustry and/or government safety standards. Being generally a largerand more capable processor than the safety processor, the systemprocessor will generally consume more power than the safety processorwhen both are active.

Similarly, when both processors are inactive, the system processor willstill consume more power than the safety processor. The system processorcan be operative to process user interface features and monitorinterface sensors (such as occupancy sensors, audio sensors, cameras,etc., which are not directly related to core safety sensing). Forexample, the system processor can direct wireless data traffic on bothhigh and low power wireless communications circuitry, accessnon-volatile memory, communicate with the safety processor, and causeaudio to be emitted from speaker 550. As another example, the systemprocessor can monitor interface sensors to determine whether any actionsneed to be taken (e.g., shut off a blaring alarm in response to a userdetected action to hush the alarm). The safety processor can beoperative to handle core safety related tasks of the hazard detector400. The safety processor can poll safety sensors (e.g., smoke, CO) andactivate alarm device 560 when one or more of safety sensors indicate ahazard event is detected. The safety processor can operate independentlyof the system processor and can activate alarm device 560 regardless ofwhat state the system processor is in. For example, if the systemprocessor is performing an active function (e.g., performing a Wi-Fiupdate) or is shut down due to power constraints, the safety processorcan still activate alarm device 560 when a hazard event is detected.

In some embodiments, the software running on the safety processor may bepermanently fixed and may never be updated via a software or firmwareupdate after the hazard detector 400 leaves the factory. Compared to thesystem processor, the safety processor is a less power consumingprocessor. Using the safety processor to monitor the safety sensors, asopposed to using the system processor to do this, can yield powersavings because safety processor may be constantly monitoring the safetysensors. If the system processor were to constantly monitor the safetysensors, power savings may not be realized. In addition to the powersavings realized by using safety processor for monitoring the safetysensors, bifurcating the processors can also ensure that the safetyfeatures of the hazard detector 400 always work, regardless of whetherthe higher level user interface works. The relatively high powerwireless communications circuitry can be, for example, a Wi-Fi modulecapable of communicating according to any of the 802.11 protocols.

By way of example, the relatively high power wireless communicationscircuitry may be implemented using a Broadcom BCM43362 Wi-Fi module. Therelatively low power wireless communications circuitry can be a lowpower Wireless Personal Area Network (6LoWPAN) module or a ZigBee modulecapable of communicating according to an 802.15.4 protocol. For example,in one embodiment, the relatively low power wireless communicationscircuitry may be implemented using an Ember EM357 6LoWPAN module. Thenon-volatile memory can be any suitable permanent memory storage suchas, for example, NAND Flash, a hard disk drive, NOR, ROM, or phasechange memory. In one embodiment, the non-volatile memory can storeaudio clips that can be played back using the speaker 550. The audioclips can include installation instructions or warnings in one or morelanguages. The interface sensors can includes sensors that are monitoredby the system processor, while the safety sensors can include sensorsthat are monitored by the safety processor. Sensors 220 and 232 can bemounted to a printed circuit board (e.g., the same board processor 210and 230 are mounted to), a flexible printed circuit board, a housing ofsystem 205, or a combination thereof.

The interface sensors can include, for example, an ambient light sensor(ALS) (such as can be implemented using a discrete photodiode), apassive infrared (PIR) motion sensor (such as can be implemented usingan Excelitas PYQ1348 module), and one or more ultrasonic sensors (suchas can be implemented using one or more Manorshi MS-P1640H12TR modules).The safety sensors can include, for example, the smoke detection chamber430 (which can employ, for example, an Excelitas IR module), the COdetection module 570 (which can employ, for example, a Figaro TGS5342sensor), and a temperature and humidity sensor (which can employ, forexample, a Sensirion SHT20 module). The power source can supply power toenable operation of the hazard detector and can include any suitablesource of energy. Embodiments discussed herein can include AC linepower, battery power, a combination of AC line power with a batterybackup, and externally supplied DC power (e.g., USB supplied power).Embodiments that use AC line power, AC line power with battery backup,or externally supplied DC power may be subject to different powerconservation constraints than battery only embodiments.

Preferably, battery-only powered embodiments are designed to managepower consumption of a finite energy supply such that hazard detector400 operates for a minimum period of time of at least seven (7), eight(8), nine (9), or ten (10) years. Line powered embodiments are not asconstrained. Line powered with battery backup embodiments may employpower conservation methods to prolong the life of the backup battery. Inbattery-only embodiments, the power source can include one or morebatteries, such as the battery pack 450. The batteries can beconstructed from different compositions (e.g., alkaline or lithium irondisulfide) and different end-user configurations (e.g., permanent, userreplaceable, or non-user replaceable) can be used. In one embodiment,six cells of Li—FeS₂ can be arranged in two stacks of three. Such anarrangement can yield about 27000 mWh of total available power for thehazard detector 400.

Referring now to FIGS. 5C and 5D, illustrated are front and rearperspective views of a speaker 550 that is electrically coupled withcircuit board 500 so as to receive instructions therefrom. Speaker 550includes a speaker body 552 and one or more mounting flanges 554 thatallow the speaker 550 to be coupled with or mounted on front casing 460.Speaker 550 also includes a plug 556 or other mounting component thatallows the speaker 550 to be electrically coupled with circuit board500. As previously described, speaker 550 may be used to audibly alertan occupant of a room within which hazard detector 400 is positioned, orto provide other messages to the occupant of the room. For example,speaker 550 may be used to alert a firefighter or other rescuerregarding the occupants remaining in the home or structure after a fireor other danger is detected or may be used to inform an occupant of asafest route out of the home or structure.

Referring now to FIGS. 6A and 6B, illustrated are front and rearperspective views of a lens button 600. Lens button 600 includes a frontsurface 602 and a rear surface 604. Lens button 600 is configured to becoupled with front casing 460 by attaching lens button 600 to light ring620, and coupling light ring 620 to a surface portion of front casing460, as shown in FIG. 4B. Lens button 600 is configured to be pressed bya user to provide input to hazard detector 400 and/or for various otherpurposes, such as quieting an alarm device. Lens button 600 is furtherconfigured to be transparent to one or more sensors positioned behindlens button 600. For example, in one embodiment, a PIR sensor ispositioned behind lens button 600. The PIR sensor is able to viewexternal objects through lens button 600 to determine if an occupant ispresent within a room in which hazard detector 400 is positioned.

The rear surface 604 of lens button 600 may have a Fresnel lens pattern606 that allows the PIR sensor, or another sensor, positioned behindlens button 600 to view far into the room in which hazard detector 400is positioned. In one embodiment, Fresnel lens pattern 606 may include aplurality of concentrically arranged rings that each provides a slightlydifferent viewing cone. Each concentrically arranged ring may provide aprogressively larger viewing area or cone than rings concentricallyarranged and located radially closer to a central axis of lens button600. In one embodiment, an internal angle of the viewing cones providedby Fresnel lens pattern 606 may vary from between about 15° and about150° so as to provide a viewing radius on a floor or wall positioneddirectly in front of the hazard detector 400 at a distance ofapproximately 10 feet or between about 0.5 m and about 8.8 m. In thismanner, the PIR sensor, or other sensor, positioned behind lens button600 may easily detect the presence of an occupant within a room in whichhazard detector 400 is positioned.

Referring now to FIGS. 6C and 6D, illustrated are front and rearperspective views of a light ring 620 that may be used to disperse lightprovided by an LED or other light source so as to provide a halo effectbehind and around lens button 600. Light ring 620 includes a bodyportion 622 and may be coupled with lens button 600 via adhesive bondingor any other method known in the art. In turn, light ring 620 may becoupled with front casing 460 such as by orienting light ring 620 withrespect to a surface of front casing 460 and pressing light ring 620axially downward relative to front casing 460 so that recessed portions625 of light ring 620 mate and couple with tabs (not shown) of frontcasing 460. These tabs may fit over the recessed portions 625 of lightring 620 and secure light ring 620 adjacent a surface of front casing460. Light ring 620 also includes a plurality of second recesses 624within which an LED (not shown) or other light source may be positionedto illuminate light ring 620. In operation, light ring 620 disperseslight provided by the LED or other light source to provide a halo effectbehind and around lens button 600.

Referring now to FIGS. 6E and 6F, illustrated are front and rearperspective views of a flexible circuit board or flex ring 640 that mayelectrically couple components positioned in front of circuit board 500,such as lens button 600, with circuit board 500. Flex ring 640 includesa tail end or ribbon 644 that may be insertable into a component ofcircuit board 500 to electrically couple lens button 600, light ring620, and/or one or more components with circuit board 500. Flex ring 640also includes a central portion that may include a PIR sensor 650 thatis positioned so as to be behind lens button 600. The central portion offlex ring 640 further includes a plurality of flanges 646 that mate withflanges (not shown) of front casing 460 so as to orient flex ring 640relative to front casing 460 and/or couple flex ring 640 therewith.Specifically, a channel 648 between flanges 646 may fit around flanges(not shown) of front casing 460 to orient and couple flex ring 640 withfront casing 460. Flex ring 640 further includes a circumferentiallyarranged ring portion 642 having a plurality of LED lights 652, or othersource of light, coupled therewith. The plurality of LED lights 652 arearranged so as to be insertable within recessed portions 624 of lightring 620. LED lights 652 illuminate light ring 620 as previouslydescribed. A bottom surface of the central portion of flex ring 640includes a pressable button 651 that is actuated as lens button 600 ispressed by a user. In this manner, input is provided to the hazarddetector 400 by the user as previously described.

As mentioned above, embodiments of the present invention, e.g., hazarddetectors 104 and 400, may be paired with an online management account.This pairing may be accomplished during the setup process for a smarthazard detector. Examples of this setup process according to the presentinvention are discussed in the next section.

Initial Background Noise Signatures

Many of the smart-home devices described above in FIGS. 1-6 can bemodified or designed, in light of this disclosure, to include one ormore sound recording devices, such as a microphone. Along with othersensors, a sound recording device can be used to ascertain the state ofthe environment within an enclosure. Sound inputs can also be used tocontrol any of the devices in the smart home system. For example, anoccupant of a home may wish to control a hazard detector using a voicecommand such as “silence the alarm.” As smart-home devices areproliferated within an enclosure, each can be designed with anintegrated or add-on sound recording device to form an interconnectednetwork of sound recording stations such that users can control anysmart home function from almost anywhere in their home. For example, auser can speak a voice command such as “turn on the dishwasher” that iscaptured by microphone of a hazard detector in the master bedroom. Thehazard detector can then interpret and/or forward the voice command tothe dishwasher or other smart appliance. In other scenarios, aninterconnected system of microphone-equipped smart-home devices can forman intercom system, an emergency notification system, a home-wide phonesystem, and/or an environment in which users can verbally command anysmart-home device or function from anywhere in their home.

When receiving voice commands or recording other sounds of interest,smart-home devices will often encounter significant levels of backgroundnoise. For example, a user asking a smart thermostat to “turn up thetemperature” may be speaking while vacuuming, while the dishwasher isrunning, or while the HVAC system is operating. These appliances andactivities can generate background noise that may cause the smart-homedevice recording the voice commands to miss or misinterpret what is saidby the user. Generally, when voice commands are misinterpreted by anyvoice-control system, users become frustrated and inpatient, oftenleading users to abandon voice control options altogether, and insteadsettling for traditional manual control techniques. In order to dealwith background noise when recording voice commands and other sounds ofinterest, the smart-home devices described herein can utilize noisecancellation algorithms combined with a background noise profiledatabase that is continuously or periodically updated.

Traditional noise cancellation algorithms generally begin at a startingstate were both background noise and foreground sounds are present, andthen gradually converge to a state where most of the background noise isbeing filtered out while the foreground sounds are allowed to passthrough the filtering operation. When a traditional noise cancellationalgorithm begins receiving sound captured by microphone, the convergenceinterval is used to distinguish between the background noise and theforeground sounds. The algorithm can determine a background noiseprofile during this convergence interval and then remove that backgroundnoise profile from the recorded sound, leaving only the foregroundsound. As used herein, foreground sounds may refer to any type of soundthat is intended to be recorded by the microphone or sound recorded inorder to control or influence the behavior of a smart-home device, suchas a voice command. Background noise a refer to any sound that is not aforeground sound.

In some embodiments, background noise may specifically refer tosteady-state noises generated by smart home appliances, such asdishwashers, air-conditioners, refrigerators, and so forth. Backgroundnoise may also refer to low-level noise such as a vacuum cleaner, atelevision in another room, sounds originating from outside of the homeenvironment, such as construction noises, airline traffic, passingtrains, and so forth. A noise signature or sound signature may refer toany information associated with a sound signal that can be used as aninput to a noise cancellation algorithm in order to attenuate anassociated background sound signal from the recording. A sound signatureor noise signature may comprise a recording of the noise or sound, afrequency analysis, a decibel level, and/or any combination of these orother characteristics of the noise or sound.

When using traditional noise-cancellation algorithms to filter outbackground noise, some existing applications are able to begin recordingbefore a voice command is given. For example, a laptop computer willgenerally receive a command from the user to activate a microphone. Uponactivation, the laptop computer can begin recording background noise andquickly determine a background noise signature before the user actuallystarts speaking to the laptop computer. Similarly, when a user intendsto use a smart phone, they will activate the phone feature, dial anumber, and position the phone next to their head before they actuallybegin speaking. The noise cancellation algorithms can use this time toascertain a background noise signature before the user's phoneconversation begins. In general, existing solutions will provide adevice an indication that sounds of interest (i.e. foreground sounds)will begin after a short delay interval. The device can then recordbackground noise before the foreground sounds begin. Thus, when usersbegin speaking the noise cancellation algorithm will have alreadyconverged, and the user will not have to endure a few seconds of soundmodulations and fluctuations in waiting for the algorithm to converge.For example, when using a smart phone to make a telephone call, thesmart phone can record background noise, such as the sound of a carengine running, identify the background noise as such, and converge thenoise cancellation algorithm before the user even says “hello.”

However, in using voice commands to control smart-home devices, thistraditional solution in certain cases may be inadequate. Most voicecommands are very short, abrupt, and provided with little or no warningto a recording smart-home device. For example, a user may say “hellothermostat, set the temperature to 75°” or “hello hazard detector,ignore the smoke.” Each of these exemplary commands may only last one ortwo seconds, which typically is not enough time for a noise cancellationalgorithm to converge. By the time the noise cancellation algorithm wasable to distinguish the background noise from the foreground sound, thecommand would be over, and the user would expect the command to beexecuted.

The embodiments described herein may be configured to detect the startof a voice command and immediately apply a background noise signature asthe starting state of the noise cancellation algorithms. The backgroundnoise signature can be selected from a plurality of background noisesignatures that have been previously collected over time. By knowing thestarting state of the background noise, the noise cancellation algorithmcan be immediately applied without having to wait for a convergenceinterval. Thus, a known background noise signature can be filtered outof even the shortest of voice commands. For example, a user may give avoice command to their hazard detector while the dishwasher is running.The hazard detector can determine of the dishwasher is running andselect the background noise signature associated with the dishwasher asa starting state for the noise cancellation algorithm without needing towait through a convergence interval to distinguish the background noiseof the dishwasher from the foreground sound during the voice command.

As will be described in greater detail below, a database of backgroundnoise signatures can be built up over time as different background noiseevents occur. Each background noise signature can be associated withattributes that can help predict when that particular background noisewill occur again. By way of example, the background noise signatureassociated with the dishwasher may be most likely to occur between thehours of 7:00 PM and 8:00 PM when the home is occupied and when thetelevision is not on. When a voice command is detected by a smart-homedevice, the smart-home device can receive an environmental input, suchas a time of day, a status of an appliance in the home, an occupancysensor input, a temperature reading, and/or the like, and automaticallyselect from the background noise signature database one or more startingnoise signatures for the noise cancellation algorithm. The entire smarthome system including the devices, the appliances, the sensors, and/orother computing devices can all work in concert to receive and provideenvironmental inputs, build the background noise signature database,capture, process, and store individual noise signatures, and capture andfilter voice commands. Some embodiments may also receive, store,process, and provide background noise signatures at a remote server,such as a cloud server, smart-home device management server, etc.

For exemplary purposes, the methods and systems for implementing abackground noise signature database will be described in relation to ahazard detector, such as the hazard detector of FIGS. 4-6. However,other embodiments are not so limited. Other embodiments may use thesenoise cancellation techniques in a thermostat, a door entry system, asecurity system, an intercom system, and/or any other smart-home device.

FIG. 7 illustrates a block diagram 700 of a smart-home devicearchitecture, according to some embodiments. This smart-home device 702may be considered a generic device that may be augmented to perform anumber of different functions. For example, the smart-home device 702may be modified to correspond to the architecture of the hazard detectordescribed above in FIGS. 1-6. The features and systems of the smart-homedevice 702 may be generically incorporated into a hazard detector, athermostat, a security system module, a surveillance system module,and/or any other smart device/appliance. For example, a thermostat,hazard detector, etc., may include a network interface 706 that includesa wireless connection 708 (e.g., 802.11, 802.15.4, etc.), a wiredconnection 710 (e.g. ethernet, coaxial cable, telephone line, DSL,etc.), and/or a cellular connection 712 for communicating with othersmart-home devices, management servers, mobile computing devices, and/orthe like.

The smart-home device 702 may include one or more environmental sensors716, such as a presence sensor 726 (e.g., PIR, microwave, etc.), atemperature sensor 730, and/or other environmental sensors 732, such aspressure sensors, light sensors, carbon monoxide detectors, and/or thelike. Although not technically a sensor per se, the smart-home device702 may also include a clock 728. The clock 728 may be implementedinternally, or the smart-home device 722 may receive a clock signal froma central server, a cellular service provider, a network connection, oranother smart-home device/appliance. These and other environmentalsensors can be used to determine what is happening around the smart-homedevice 722 when recording begins. For example, the clock 728 can providea time-of-day input and the presence sensor 726 can provide an occupancystatus, and these can be used to select a background noise signature tobe used by the noise cancellation algorithm when the recording begins.

Additionally, the smart-home device 702 may include a microphone 704that may be used to capture recorded sound used by the noisecancellation algorithm. For embodiments involving smart-home devicesthat would normally include a microphone, the microphone 704 may simplypiggyback on the existing architecture. For example, an intercom systemwill have modules equipped with microphones, and the embodimentsdescribed herein can use these existing microphones for noisecancellation. In another example, a thermostat might not typicallyinclude a microphone, and thus the thermostat can be modified to includethe microphone 704 to accept voice commands and perform noisecancellation. In some embodiments, the microphone 704 can remain in aperpetually active state such that it is constantly detectingsurrounding sounds. The microphone 704 can be coupled to a processingsystem 714 that analyzes the surrounding sounds and detects verbalcommands or other sound signatures that can be used to generate aresponse from the smart-home device 702. For example, the processingsystem 714 can analyze incoming sounds to identify phrases such as“Hello Nest,” or “Voice Command” and then respond accordingly. Such acommand may trigger the processing system 714 to begin recording soundand/or performing a noise cancellation algorithm. In some embodiments,the microphone 704 can activate in response to inputs received fromother environmental sensors 716. For example, the microphone 704 canbecome active and record sounds when the presence sensor 726 determinesthat an enclosure is occupied or that a user is within the immediatevicinity of the smart-home device 702. Other inputs can be used toactivate microphone 702, such as a manual user input (e.g. pressing abutton on a thermostat or hazard detector), controlling the smart-homedevice 702 with a smart phone, or performing a physical hand or armgesture in view of the smart-home device 702. In some embodiments themicrophone 704 can be activated in response to an alarm condition orother detected environmental condition by the smart-home device 702 orother smart-home devices in the enclosure. For example, microphone 704may be activated in response to a smoke alarm in order to accept voicecommands for silencing a hazard detector, contacting an emergencyresponse service, and/or the like.

As described above, the smart-home device 702 may include a processingsystem 714 configured to receive sounds detected by the microphone 704and perform one or more noise cancellation algorithms. The processingsystem 714 may include a noise cancellation module 718 insoftware/hardware to perform such algorithms. It will be understood thatthe processing system 714 may include a dedicated processing system forthe microphone 704 and a noise cancellation algorithm. In otherembodiments, the processing system 714 may comprise a microprocessor ormicrocontroller that is programmed to perform other operations, such ascommunicating through the network interface 706, polling and analyzingreadings from the environment sensors 716, controlling environmentalsystems, such as an HVAC system, and/or the like. The processing system714 may be split among one or more microprocessors and possibly mountedto different circuit boards within the smart-home device 702, such as aback plate circuit board and/or a head unit circuit board.

Smart-home device 702 may also include one or more memories 720 thatinclude a sound signature data store 722 and a sound attribute datastore 724. The sound signature data store 722 may be configured to storea plurality of sound signatures that can be accessed and retrieved asthe background noise signature for the noise cancellation algorithms.Each sound signature may be associated with one or more soundattributes. These attributes may be used to determine when a particularsound signature should be selected as the initial background noisesignature. For example, a sound signature corresponding to arefrigerator compressor may have attributes that describe a time of daywhen the refrigerator compressor is most likely to be active, an inputcondition that may be received from the refrigerator itself when thecompressor is active, and/or the like. Each sound signature may beassociated with one or more sound attributes, and the sound attributesmay be stored in a separate data store as illustrated by FIG. 7, or inthe same data store as the sound signatures themselves.

The arrangement of modules and functions as illustrated in FIG. 7 ismerely exemplary and not meant to be limiting. Other embodiments may addadditional modules, such as additional environmental sensors, additionalnetwork communication options, different memory configurations, anddistributed processing systems. Therefore, it will be understood thateach module in FIG. 7 may be combined with other modules or subdividedinto additional sub-modules as needed. Additionally, each module may beimplemented in hardware, software, or a combination ofhardware/software.

FIG. 8 illustrates a flowchart 800 of a method for building a databaseof stored sound profiles, according to some embodiments. The method mayinclude beginning a learning interval (802). In some embodiments, thelearning interval may correspond to a time interval that beginsfollowing the installation of the smart-home device and lasts for apredefined number of days or weeks. For example, the learning intervalmay begin upon installation and last for an interval of three weeks. Insome embodiments, the learning interval may be instituted periodicallythroughout the lifetime of the smart-home device. For example, alearning interval may begin each night at midnight and last until 6:00AM. In embodiments where sound is continuously recorded by a microphoneon the smart-home device, the learning interval may be extracted fromrecordings occurring immediately prior to receiving a voice command.Once a voice command is recognized, the sounds in the previous 30seconds, one minute, 90 seconds, five minutes, etc., may be analyzed asa “learning interval.” In some embodiments, the learning interval maybegin when a presence sensor on the smart-home device—possibly inconjunction with present sensors in other smart-home devices—determinethat the enclosure is unoccupied, thereby reducing the chance ofinterfering background noise caused by human occupants.

The method may also include detecting a steady-state background noise(804). Some embodiments may limit their sound signature database tosteady-state sound patterns that will be substantially the samethroughout the recording interval. For example, appliances such asrefrigerators, HVAC systems, fluorescent lights, computer fans,microwave ovens, and so forth, will generally produce a somewhatconsistent steady-state sound during their operation. When using thesetypes of steady-state sounds as the initial background noise signaturefor a noise cancellation algorithm, the algorithm will have very littlework to do in order to converge and produce relatively noise-free sound.Other embodiments may also analyze and record more transient soundsignatures. These may include sounds such as television noise, a vacuumcleaner, low-level radio sounds, construction noise, power tools, and soforth. These more transient sound signatures may also be useful insituations where voice commands that are received are relatively short.For example, a voice command such as “increase the temperature by 5°”may only take two or three seconds to capture, and a transient noise,such as a drill or a blender, will be relatively steady-state during theshort interval of the voice command.

During a learning interval, background noises may be detected bypassively listening to the surrounding environment and identifyingsounds that are persistent, repetitive, and/or loud enough to beconsidered a potential source of background noise interference. Forinstance, the noise of blender could be detected two or three differenttimes during a learning interval. On the third repetition, the methodcould identify the noise of the blender as a potential background noisesource. In another example, a microwave oven could begin operating, andafter 10 seconds, 15 seconds, 20 seconds, etc., the method couldidentify the sound of the microwave oven as a potential background noisesource.

The method may also include recording background noise samples (806).The background noise samples may be recorded by the microphone on thesmart-home device and stored locally in a memory on the smart-homedevice. Sound processing algorithms can be used to isolate thebackground noise in the recording. For example, if the sound associatedwith a refrigerator compressor is being recorded, the smart-home devicecan filter out other noise, such as human voices, doors closing, dishesbeing loaded into a dishwasher, and so forth.

The recorded background noise samples may be recorded and/or stored inmany different formats depending on the particular embodiment. In oneembodiment, the actual recording of the background noise sample may bestored in a one second clip, a two second clip, a five second clip,and/or the like. In other embodiments, a frequency analysis may beperformed, and the dominant frequencies of the recorded background noisesample can be stored. In other embodiments, the recorded backgroundnoise can be used to select from a plurality of pre-recorded backgroundnoise signatures such that the newly recorded background noise does notneed to be stored.

In some embodiments, a management server in communication with thesmart-home device can store sound signatures for common householdappliances. The recorded background noise can be sent to the managementserver, which can then determine the household appliance with theclosest match to the recorded sound. For example, the smart-home devicecan record the sound of a dishwasher. However, the recording of thedishwasher may include other background noise sounds, such as voices orrunning water, and the microphone of the smart-home device may not belocated in a position that is ideal for capturing the sound of thedishwasher. The recorded dishwasher sound or sound signature can betransmitted to the management server, and the management server candetermine that the dishwasher is most likely a particularmake/model/year and retrieve a corresponding pre-stored sound signaturefor that particular dishwasher. This signature can then be transmittedto the smart-home device for use in noise cancellation. This type ofembodiment offers the advantage that background noise signatures can berecorded in isolation by a manufacturer of the smart-home device withoutneeding to exclude other background noises. In some embodiments, themanagement server can receive recordings from a number of differentsmart-home devices in different homes. The management server can thenselect the best representative sample of background noise. For example,the management server could receive recordings of 10 differentdishwashers of the same make/model/year. The management server couldthen select the recording having the best sound quality. That recordingcan then be transmitted to the 10 different smart-home devices indifferent homes and used for noise cancellation each.

In some embodiments, a particular smart-home device can make multiplerecordings of the same background noise and select the recording withthe highest quality. For example, the hazard detector located in akitchen could make five different recordings of a refrigeratorcompressor at different times of day. After analyzing the five differentsound recordings, the smart-home device can select the recording madeduring an unoccupied time interval that has the fewest number ofinterfering sounds.

The method may also include recording sound attributes for the capturedbackground noise samples (808). As described briefly above, soundattributes can be any data that characterizes when the background noisesample is likely to occur again in the future. By way of example, asound attribute may include an indication from a smart home appliancethat the appliance is operating. For instance, a refrigerator maygenerate a signal indicating that the refrigerator compressor has turnedon. This attribute may be used to determine that the recorded sound islikely the sound of the refrigerator compressor, and that thisparticular recorded background noise signature should be used when sucha signal is generated by the refrigerator. Sound attributes may alsoinclude a time of day, a probability of occupancy, a temperaturemeasurement, an input received from a user directly at the smart-homedevice or through another computing device, and so forth. An attributemay also be assigned by user input. For example, a user may indicatethat a refrigerator is running, a vacuum cleaner is operating, a fan isrunning, etc., through a user interface of the smart-home device.Attributes may also include smart appliance schedules received fromother smart-home devices. For example, a smart thermostat may include aschedule of when the HVAC system will be operating. The schedule can beshared with other smart-home devices in the enclosure, and thesmart-home devices can utilize the shared schedule to determine when abackground noise signature corresponding to the HVAC system should beselected for noise cancellation.

The stored sound attributes can be used to establish one or moreenvironmental contexts during which each identified background noisesample is most likely to occur. As will be described in greater detailbelow, when starting a noise cancellation algorithm, one or moreenvironmental inputs can be received that correspond to differentattributes associated with the background noise signatures. Attributevalues can be compared (e.g., the current time of day can be compared toan attribute time of day) in order to select one or more backgroundnoise samples that are most likely to be occurring when the noisecancellation algorithm is initialized.

The method may continue until the learning interval is concluded (810).In order to adapt to changing environmental circumstances, to accountfor new appliances in the enclosure, and/or to generally improveperformance, the smart-home device can continuously refine backgroundnoise signatures and their corresponding attributes over time (812).Although the current learning interval may have concluded, futurelearning intervals may be carried out in the periodically or as needed.For example, a learning interval from 5:00 PM to 6:00 PM may havecaptured a number of background noise signatures with correspondingattributes. However, because this time of day is generally busy when thehome is occupied, the sound signatures and corresponding attributes maynot be as accurate as they could be. A future learning interval may beinstituted on a day between 5:00 PM and 6:00 PM when the smart-homedevice determines that the enclosure is likely unoccupied.

FIG. 9 illustrates a diagram 900 of a home with one or more smart-homedevices 908, 916 that capture background noise signatures 910, 914, 918,according to some embodiments. The method of flowchart 800 describedabove can be carried out by a single smart-home device. Alternatively,the method of flowchart 800 and other methods may be carried outcooperatively by a network of smart-home devices within a home, or evenwithin a group of homes communicating through the Internet. In thissimplified example, enclosure 902 can include a first room 904 and asecond room 906 that may or may not be adjacent. For example, the firstroom 904 may include a smart-home device in the form of the hazarddetector 908 and appliances such as a washing machine 912. The secondroom 906 may include a smart-home device in the form of a thermostat916, as well as appliances such as a refrigerator 922 and a microwaveoven 920.

Each appliance may generate its own unique sound signature, and thesound signatures can be captured by any of the available smart-homedevices. For example, based on their proximity to the appliances, thehazard detector 908 can capture the sound signature 910 from the washingmachine 912. Similarly, the thermostat 916 can capture the soundsignature 918 from the microwave oven 920. Assuming that therefrigerator 922 is against a wall separating the first room 904 fromthe second room 906, either the hazard detector 908 or the thermostat916 may be best suited to capture the sound signature 914 of therefrigerator 922. In some embodiments, both the hazard detector 908 andthe thermostat 916 can capture the sound signature 914 of therefrigerator 922. Both versions of the sound signature 914 can then becompared and the highest-quality version could be used by bothsmart-home devices. In some embodiments, the noise from occupants andother appliances in the second room 906 may make it difficult for thethermostat 916 to capture a high-quality version of the sound signature914 of the refrigerator 922. For example, a kitchen may be frequentlyoccupied and may have many other background sounds and other appliancesthat are operating at the same time. Therefore, the hazard detector 908may be better suited in the first room 904 to capture the soundsignature 914 of the refrigerator 922, as the compressor of therefrigerator 922 may propagate through the wall while the otherbackground noises in the second room 906 do not.

In some embodiments, each smart-home device may be primarily responsiblefor capturing sound signatures at their respective locations. Becauseappliances may sound differently based on location of the smart-homedevice, some noise cancellation algorithms may be more effective whenusing a background noise signature captured from the location of thesame microphone used in the sound recording for which the noisecancellation is taking place. By way of example, the hazard detector 908may at times be susceptible to background noise from the sound signature910 of the washing machine as well as the sound signature 914 from therefrigerator 922. However, because the sound signature 914 of therefrigerator 922 propagates through the wall of the enclosure 902, itmay be more muffled or attenuated than it would sound according to themicrophone of the thermostat 916. Therefore, in some cases, the soundsignatures 910, 914, 918 may be centrally stored and shared betweensmart-home devices, while in other cases, certain sound signatures maybe specific to a particular smart-home device. Some embodiments may testany available sound signatures available to a specific smart-home devicefor an appliance, and choose the one that causes the noise cancellationalgorithm to converge the quickest. For example, the hazard detector 908can use its own version of the sound signature 914 of the refrigerator922, as well as the version available from the thermostat 916. Aftertesting both of these versions of the sound signature 914, the hazarddetector 908 can select the version of the sound signature 914 thatcauses its noise cancellation algorithms to converge the fastest.

FIG. 10 illustrates a diagram 1000 of smart-home devices in multiplehomes 1008, 1014 in communication with a management server 1002,according to some embodiments. As described above, background noisesignatures 1010 can be stored locally in a first home 1008 on one ormore smart-home devices or on a local computer system. These soundsignatures 1010 can be stored with sound attributes 1012 that arespecific to the sound signatures 1010 within the first home 1008. Thesesound signatures 1010 and sound attributes 1012 can be locally capturedin the first home 1008 and can be specific to the environment of thefirst home 1008. For example, the sound signatures 1010 may include thesound of the local HVAC system, and the sound attributes 1012 mayinclude characteristics describing when the local HVAC system is likelybe active.

The smart-home device network of the first home 1008 may be incommunication with a remotely located management server 1002. Themanagement server 1002 may also store sound signatures 1004 and soundattributes 1006. In some embodiments, the sound signatures 1004 and thesound attributes 1006 may include backup copies of the sound signatures1010 and sound attributes 1012 stored at the first home 1008. In someembodiments, the sound signatures 1004 may include a relatively largerdatabase of known sound signatures for different appliances. Forexample, the sound signatures 1004 may include sound signatures fordifferent makes/model/years of different brands of refrigerators. Thiscan allow a user to enter information regarding appliances in their homevia a user interface of smart-home devices in order to select apre-existing sound signature for those appliances. This may eliminatethe need for the local smart-home device to record and analyzebackground sounds. Instead, the sound signatures can be downloaded fromthe management server 1002. Instead of receiving appliance informationfrom a user interface, smart appliances may also be able to communicatewith the smart-home device network and provide self-identifyinginformation. This information can then be sent to the management server1002 where appropriate background noise signature for the appliances maybe retrieved and sent back to the smart-home device network of the firsthome 1008.

Sound attributes 1006 stored at the management server 1002 may be moregeneral in nature than the sound attributes 1012 stored at the firsthome 1008. While the sound attributes 1012 stored at the first home 1008may be specific to the sound signatures 1010 of the first home 1008(e.g., when does the local HVAC system usually operate?), The soundattributes 1006 at the management server 1002 may be generallyapplicable to the sound signatures 1004 in many different homes. Forexample, a sound signature for refrigerator stored in the soundsignatures 1004 at the management server 1002, may include in the soundattributes 1006 information descriptive of when refrigerators in generalare usually active. Certain makes/models/years of refrigerators mayrequire the compressor to operate more or less often than others, andthis information can be used to determine a probability as to when eachmake/model/year refrigerator may be active in a home. The soundattributes 1006 at the management server 1002 can be derived frominformation received from multiple homes using smart-home devices. Forexample, the first home 1008 and a second home 1014 may both have thesame brand of refrigerator. The first home 1008 and the second home 1014may both transmit sound attribute information to the management server1002 indicating when the refrigerator compressor is most likely to beactivated. This information (and similar information from many otherhomes) may be used to generate a general usage attribute for soundsignatures related to that particular make/model/year of refrigerator.

The sound attributes 1006 at the management server 1002 may then be usedto inform the decision-making process of local smart-home devices invarious homes. For example, upon initial setup, and possibly prior to orduring a learning interval, smart-home devices in the second home 1014may receive sound attributes 1006 from the management server 1002 as astarting point. These may be used until the smart-home devices in thesecond home 1014 are able to go through a learning interval anddetermine sound attributes 1018 and sound signatures 1016 for any localappliances. During a learning interval, sound signatures 1016 capturedby a local smart-home device, as well as sound attributes 1018determined by the local smart-home device may be used to adjust orreplace sound attributes 1006 and/or sound signatures 1004 received fromthe management server 1002 as a starting point. For example, soundattributes 1006 received from the management server 1002 may indicatethat a dishwasher is generally used between the hours of 6:00 PM and8:00 PM. However, occupants of the second home 1014 may follow adifferent schedule. On setup, the local smart-home devices in the secondhome 1014 can download sound signatures and sound attributes for themake/model/year of dishwasher from the management server 1002. During alearning interval, the smart-home devices may adjust the sound signatureto more closely match the sound captured by the local smart-homedevices, and may also adjust the sound attributes 1018 to more closelymatch the schedule of the occupants of the second home 1014.

Sound signature and sound attribute information may be shared betweenhomes based on their geographic proximity. The management server 1002will generally know the location of the first home 1008 and/or thelocation of the second home 1014. Other occupant-specific informationmay also be made available to the management server 1002, such as thenumber of occupants, an occupancy schedule, energy usage information,demand-response preferences, and/or the like. This information can beused by the management server to determine whether it would bebeneficial to share sound signature and sound attribute informationbetween the first home 1008 and the second home 1014. For example, ifthe occupants of the first home 1008 and the occupants of the secondhome 1014 generally follow a similar occupancy schedule, then it ispossible that the sound attributes 1012, 1018 of the two homes 1008,1014, respectively, will be similar, and these attributes can be sharedthrough the management server 1002.

Storing sound signatures 1004 and/or sound attributes 1006 at themanagement server 1002 may provide additional benefits besides sharingsound signatures and attributes between homes. In some embodiments, thisinformation can be used to diagnose a state-of-health for localappliances. For example, the management server 1004 include a soundsignature for a healthy refrigerator of a particular make/model/yearinstalled in the first home 1008. Smart-home devices in the first home1008 may capture and analyze a sound signature of the particularrefrigerator installed in the first home 1008. The locally capturedsound signature can then be compared to the healthy sound signaturestored at the management server 1002. If there is significant deviation,this can be used to determine that this particular refrigerator may bemalfunctioning or not operating optimally. An indication can then beprovided to a user by a smart-home device that their refrigerator may bemalfunctioning. It may include specific advice for a particular type ofappliance (e.g., “you may need to clean your refrigerator coils”). Insome embodiments, a state-of-health report can be generated for all ofthe known appliances in a home based on their sound signatures, and thisreport can be accessible through the user interface of the localsmart-home devices.

FIG. 11 illustrates a chart 1100 of how sound profiles can be combined,according to some embodiments. Often within an enclosure, more than onebackground noise will be present when a noise reduction algorithm isrun. For example, a voice command may be received by a smart-home devicewhile multiple appliances, such as a refrigerator compressor and adishwasher are operating at the same time. The smart-home device canhandle this situation in a number of different ways. In one embodiment,sound signatures and sound attributes can be captured and stored on anindividual basis. In chart 1100, each sound signature on the horizontalor vertical axis represents a sound signature for a single type ofbackground noise. For example, the refrigerator compressor would includeits own sound signature 1104 and set of corresponding sound attributes.The dishwasher would also include its own sound signature 1102 and setof corresponding sound attributes. During a recording session, each setof sound attributes may be analyzed individually to determine whichappliances are likely to be active at that time. If it is determined,based on the individual attributes, that both the dishwasher and therefrigerator compressor are likely to be operating, then the soundsignature 1102 of the dishwasher and the sound signature 1104 of therefrigerator compressor can both be retrieved and used by anoise-cancellation algorithm. Some noise-cancelation algorithms mayaccept more than one background noise signature as a starting point, inwhich case both the sound signature 1102 and/or the sound signature 1104can be provided to the noise-cancelation algorithm. In cases where thenoise-cancelation algorithm accepts only a single background noisesignature, the sound signature 1102 of the dishwasher and the soundsignature 1104 of the refrigerator compressor can be combined to form acomposite sound signature 1106 for both appliances. The manner in whichtwo sound signatures can be combined will depend on the format of thestored sound signatures. For example, if the sound signatures comprise aset of frequency components, then the union of the frequency componentsof sound signatures 1102 and 1104 can be used as the composite soundsignature. Sound signatures comprising a recording of the actual soundcan be convolved together to form a composite sound, and so forth.

Chart 1100 illustrates how the composite sound signature 1106 is relatedto the source sound signatures 1102, 1104. In some embodiments, soundsignatures may be combined and/or stored as needed. For example, if thedishwasher and the refrigerator compressor are determined to likely beactive during a recording session, the composite sound signature 1106can be generated and stored at that time and be made available forfuture use. In some embodiments, a memory structure similar to chart1100 may be stored locally or remotely. This memory structure maypre-calculate combinations of individual sound signatures. This mayallow combinations of sound signatures to be retrieved without needingto perform such a combination at the time the recording session isdetected. Combining and generating, for example, a database of soundsignature combinations may be performed locally at the smart-home deviceor remotely at the management server. For example, a local smart-homedevice may record sound signatures for 10 different appliances. These 10sound signatures can be sent to the management server, where they arecombined in various ways. The sound signature combinations can betransmitted back to smart device for future use. This may beadvantageous in cases where the smart-home devices operate on a strictpower budget and/or where they use relatively low-power microprocessors.In practice, each individual sound signature can be less than 1 kB size,so storing a database of sound combinations will typically fit on thememory of the local smart-home devices.

Although chart 1100 is illustrated as being two-dimensional in nature,other embodiments may include additional dimensions of soundcombinations. For example, a smart-home device may determine that it islikely that three or more appliances are operating simultaneously. Inthis case, the smart-home device can determine a composite soundsignature for the three or more sound signatures. As described above,this information can be pre-calculated or calculated on-the-fly asneeded.

FIG. 12 illustrates a flowchart 1200 of a method for selecting a storedsound profile for a noise-reduction routine, according to someembodiments. The method may include using a learning interval to buildup a database of sound signatures (1202), and/or sharing soundsignatures between a local smart-home device and a remote managementserver (1204). These steps may be carried out as described above.Additionally, the learning interval may be used to associate soundsignature attributes with each of the sound signatures to determinetimes when it is most likely that sound sources associated with eachsound signature are active.

The method may also include determining that a non-background soundshould be recorded (1206). In some embodiments, the non-background soundmay correspond to a voice command. Such a voice command may be initiatedby a key phrase, such as “Hello Nest.” In other embodiments, thesmart-home device can automatically begin recording each time it detectsthe sound of a human voice. A determination that a non-background soundshould be recorded may also be made based on other types of user inputs.For example, a user may press a button on the smart-home device. A usermay also perform a hand gesture that is detected by a motion sensor onthe smart-home device. The smart-home device may also respond toenvironmental conditions that may start a recording session. Forexample, if a hazard detector detects a hazard condition, such as smoke,the hazard detector may begin listening for voice commands instructinghazard detector to call an emergency response team or indicating a falsealarm situation. Smart-home device may also use occupancy sensors tobegin recording when a user approaches the smart-home device asdetermined by the occupancy sensor. For example, a PIR sensor on athermostat or hazard detector may indicate when a user is approachingthe smart-home device, and the device may begin listening for voicecommands.

The method may also include receiving an environmental input (1208). Asdescribed above, the environmental input may include any input orinformation that may be used to select existing sound signatures forsound sources that are likely to be active. The environmental input mayinclude a time of day received from a local clock timer or from aninternal processor clock, an indication of occupancy, a statusindication from a local smart-home appliance, and/or the like. In someembodiments, the environmental input may also include a brief recordingof sound prior to the beginning of the recording session. For example,as a user approaches the smart-home device, the occupancy sensor mayindicate that the microphone should begin recording sound. For a fewseconds before the user begins to speak, the microphone may record thesound of a dishwasher. This recording can then be compared to the localsound signatures, and the sound signature of the dishwasher can be usedas the background noise signature for a noise cancellation algorithmthat begins operating when the user start speaking Some embodiments mayoperate primarily according to a time-of-day schedule. In otherembodiments, the environmental input may include a plurality ofdifferent inputs that are analyzed together in comparison to the soundattributes.

The method may further include selecting a stored sound signature usingthe environmental input (1210). The sound signature may be selected fromthe locally stored sound signatures or may be retrieved from amanagement server. The sound signature may be a composite of multipleindividual sound signatures as described above. The sound signature maybe selected based on a comparison between the attributes of the storedsound signatures and the environmental inputs. The environmental inputscan be compared to the attributes for each stored sound signature and ascore can be generated for each sound signature. Each sound signaturewith the score above a predetermined threshold may be selected as asound signature to be used in generating the background noise signaturefor the noise cancellation outer. For example, if the environmentalinputs include a time of day and an occupancy indication, these may becompared to an occupancy attribute and a time of day schedule for eachsound signature. It may be that a vacuum and a microwave oven are mostlikely to be used when the house is occupied at the current time,whereas the HVAC system in the refrigerator are more likely to beoperating at a different time or according to a different occupancypattern. The microwave and the vacuum can be assigned a score for eachapproximate match between a sound signature attribute and anenvironmental input. Scores may be weighted based on importance, e.g.,matching the time of day attribute maybe twice as important as matchingan occupancy attribute. A total score for each background noisesignature may be generated and compared to a threshold. For example, ifthere is greater than a 50% match between attributes and environmentalinputs, then the corresponding sound signature can be selected to bepart of the background noise signature for the noise cancellationalgorithm. This threshold may be adjusted dynamically over time asattributes for each sound signature are refined and feedback is receivedfrom the noise cancellation algorithm as described in greater detailbelow. If more than one sound signature is selected, then a compositesignature can be generated/selected for use as the background noisesignature. In embodiments where composite signatures have beenpre-calculated, the environmental inputs can be compared to theattributes associated with the composite signature. Instead of selectingmultiple individual sound signatures, these embodiments can select thesound signature with the highest score, be it an individual soundsignature or a composite sound signature.

The method may additionally include using the selected sound signatureas the initial state in the noise cancellation algorithm (1212). Manydifferent noise cancellation algorithms may accept an initial backgroundnoise signature. Selecting/using one of these noise cancellationalgorithms would be within the knowledge of one having ordinary skill inthe art, and thus the operation of noise canceling algorithms is beyondthe scope of this disclosure. As mentioned previously, the initial stateprovided to these noise-cancellation algorithms those in limited to thebackground noise recording taking place just prior to the operation ofthe noise cancellation algorithm. For example, when using a smart phone,the phone would previously begin recording background noise as the userenters/selects a telephone number. This recording itself would then beused as the starting state for the noise cancellation algorithm duringthe phone call. In contrast, the embodiments described herein receiveenvironmental inputs to determine a background noise sound that is mostlikely to be occurring, and then use the environmental input to selectamong a plurality of pre-existing background noise signatures.

The method may also include repeating steps 1206 through 1212 for eachnon-background sound detected (1214). As each sound recording sessionends, the smart-home device can generate diagnostic information for howwell be selected background noise signature matched the actualbackground noise in the noise cancellation algorithm. For example, thesmart-home device can determine a convergence interval length, i.e., adetermination of how long it took the noise cancellation algorithm toconverge from the starting point of the selected background noisesignature. This information can be used to refine the selected soundsignature as well as the accompanying attributes. For example, if asound signature corresponding to a dishwasher was selected as thebackground noise signature, but it took more than a threshold amount oftime for the noise cancellation algorithm to converge (e.g. 500 ms),then the sound attributes may be adjusted to lessen the likelihood thatthe dishwasher sound is selected during similar environmental conditionsin the future. Additionally, it may be the case that the dishwasher wascorrectly selected, but that the sound signature is low quality. Thesound signature for the dishwasher can be flagged, and better soundsignatures can be recorded/generated during future learning intervals toreplace the existing sound signature such that the noise cancellationalgorithm will converge faster in the future.

FIG. 13 illustrates a flow diagram 1300 of a converging noise-reductionroutine, according to some embodiments. Similar to the architecturedescribed in FIG. 7, the smart-home device may include a noisecancellation module 718 and a memory 720. Near the beginning of arecording interval, an environmental input 1306 may be received and usedto select a sound signature 1304 from a plurality of sound signatures722. The selected sound signature 1304 may represent the initialbackground noise state 1302 for use by the noise cancellation module718.

In some embodiments, additional environmental inputs may also bereceived in addition to the first environmental input 1306 used toselect the initial background noise state 1302. For example, after thenoise cancellation algorithm begins to filter noise from the recordinginterval, the smart-home device can continue receiving status inputsfrom smart appliances, recording ambient background noise in betweenwords spoken by a user, and so forth. These initial inputs can be usedto further characterize the types of sounds that are surrounding anintended voice recording. After the recording interval begins, a newsmart home appliance may become active. These additional inputs can bereceived by the smart-home device and it may be determined that theinitial background state 1302 needs to be updated.

Some embodiments may also receive feedback from the noise cancellationmodule 718 that indicates that the selected sound signature 1304 is notan accurate representation of the actual background noise recorded bythe microphone of the smart-home device. Metrics may be received thatindicate an approximate time for convergence of the noise cancellationalgorithm. If the estimated convergence time is more than apredetermined threshold (e.g., 500 ms, one second, 1500 ms, etc.), thenthe noise cancellation module 718 may be provided a new background noisesignature such that it can converge faster.

The noise cancellation module 718 can generate feedback and/oradditional environmental inputs that can be used to select a new soundsignature 1308 from the memory 720 to be used as the updated backgroundnoise state 1310. Some embodiments may analyze the convergence of thenoise cancellation algorithms periodically (e.g. everyone 100 ms, every500 ms, etc.) and select a new sound signature 1308 such that theconvergence of the noise cancellation algorithm can be accelerated.Typically, a new background noise signature that more accuratelyrepresents the true background noise will cause the noise cancellationalgorithm to converge faster than it would to continue letting the noisecancellation algorithm converge using a less accurate initial backgroundnoise signature.

FIG. 14 illustrates a timeline 1400 of stored sound profiles, accordingto some embodiments. Some embodiments may construct a sound signatureschedule that can be used as at least an initial means for selectingpossible sound signatures to use as the background noise signature.Timeline 1400 illustrates how such a sound signature schedule may berepresented for a 24 hour period for one week intervals. Each week,actual environmental inputs that indicate when certain sound signaturesare present in the surrounding environment can be used to update thesound signature schedule.

Once the sound signature schedule is established, the smart-home devicecan use the sound signature schedule as a starting point for selectingan initial background signature during a recording interval. Forexample, every day at 6:00 AM a smart thermostat may turn on the HVACsystem for the home. The sound signature schedule includes an entry fora sound signature corresponding to the HVAC system every day at 6:00 AMfor at least the one hour interval. If the recording interval occursduring this time, the smart-home device can check the sound signatureschedule and quickly determine that the sound signature corresponding tothe HVAC system should very likely be selected as at least one of thecomponents of the background noise signature. This can eliminate needingto test other environmental inputs against attributes of the HVAC thesystem sound signature, and the smart-home device can instead testenvironmental inputs against the attributes of other sound signatures todetermine if any other appliances or noise sources may be active inaddition to the HVAC system.

Some embodiments may establish a schedule similar to timeline 1400 usingdifferent environmental inputs. For example, a probability mapping canbe established for an occupancy sensor in relation to existing soundsignatures. The probability mapping can, for each sound signature,indicate whether the sound signature is correlated with, not correlatedwith, or independent of, an occupancy indication. Other environmentalinputs may also have their correlation with sound signatures similarlypre-calculated for efficient lookup when determining a background noisesignature.

FIG. 15 illustrates a scenario 1500 for communication of sound profilesbetween homes, according to some embodiments. Occasionally, certainbackground noise sounds may occur on a very infrequent or one-timebasis. For example, construction noise, loud traffic (e.g. a snowplow ortractor-trailer), aircraft noise, and/or the like may be occurring in aneighborhood. In the example of FIG. 15, a loud truck 1502 may betraveling down a road adjacent to a plurality of different homes 1504,1506, 1508, 1510. Because the noise of the truck 1502 may not have beenpreviously recorded by any smart-home device in any of the homes 1504,1506, 1508, 1510, the sound would not be available as a pre-stored soundsignature for use as an initial background noise signature in any of theindividual homes.

However, if smart-home devices in any of the homes are connected to acentral management server 1512, this network capability may be leveragedsuch that one smart-home device can send a sound signature to additionalhomes 1508, 1510 for use in noise canceling algorithms. As a soundsource moves through a neighborhood, a first home 1504 can detect,record, and/or analyze the sound to generate a sound signature. Thesound signature can be transmitted to the management server 1512. Themanagement server 1512 can then transmit the sound signature tosurrounding homes 1508, 1510 along with an indication that the soundsource is nearby. If a recording interval is initiated in thesurrounding homes 1508, 1510, then the indication that the sound sourceis nearby can be used as an environmental input to select the soundsignature transmitted from the management server 1512 for use in thenoise cancellation algorithms.

For example, as the truck 1502 moves through the neighborhood, the soundof the truck may be recorded by the first home 1504, and the first homemay generate a corresponding sound signature that is transmitted to themanagement server 1512. The management server 1512 can then determine ifthere are any nearby homes 1508, 1510 that may be also be in range ofthe path of the truck 1502. The management server 1512 may use Internetmapping services to determine which homes with registered smart-homedevice networks may be in the path of the truck 1502. The managementserver 1512 can then transmit the sound signature provided by the firsthome 1504 to the surrounding homes 1508, 1510. In some embodiments, themanagement server 1512 may also identify homes 1506 that, althoughwithin the immediate vicinity of the first home 1504, would likely notreceive the sound signature from the management server 1512 in time touse the sound signature as an initial background noise signature in anoise cancellation algorithm. If a smart-home device in home 1510 beginsreceive a voice command from a user as the truck 1502 passes by the home1510, the smart-home device can then utilize the received soundsignature of the truck 1502 recorded by the first house 1504 andimmediately begin filtering the noise of the truck 1502 from therecording.

Referring next to FIG. 16, an exemplary environment with whichembodiments may be implemented is shown with a computer system 1600 thatcan be used by a user 1604 to remotely control, for example, one or moreof the sensor-equipped smart-home devices according to one or more ofthe embodiments. The computer system 1610 can alternatively be used forcarrying out one or more of the server-based processing paradigmsdescribed hereinabove or as a processing device in a larger distributedvirtualized computing scheme for carrying out the described processingparadigms, or for any of a variety of other purposes consistent with thepresent teachings. The computer system 1600 can include a computer 1602,keyboard 1622, a network router 1612, a printer 1608, and a monitor1606. The monitor 1606, processor 1602 and keyboard 1622 are part of acomputer system 1626, which can be a laptop computer, desktop computer,handheld computer, mainframe computer, etc. The monitor 1606 can be aCRT, flat screen, etc.

A user 1604 can input commands into the computer 1602 using variousinput devices, such as a mouse, keyboard 1622, track ball, touch screen,etc. If the computer system 1600 comprises a mainframe, a designer 1604can access the computer 1602 using, for example, a terminal or terminalinterface. Additionally, the computer system 1626 may be connected to aprinter 1608 and a server 1610 using a network router 1612, which mayconnect to the Internet 1618 or a WAN.

The server 1610 may, for example, be used to store additional softwareprograms and data. In one embodiment, software implementing the systemsand methods described herein can be stored on a storage medium in theserver 1610. Thus, the software can be run from the storage medium inthe server 1610. In another embodiment, software implementing thesystems and methods described herein can be stored on a storage mediumin the computer 1602. Thus, the software can be run from the storagemedium in the computer system 1626. Therefore, in this embodiment, thesoftware can be used whether or not computer 1602 is connected tonetwork router 1612. Printer 1608 may be connected directly to computer1602, in which case, the computer system 1626 can print whether or notit is connected to network router 1612.

With reference to FIG. 17, an embodiment of a special-purpose computersystem 1700 is shown. For example, one or more intelligent components,processing engine 206 and components thereof may be a special-purposecomputer system 1700. The above methods may be implemented bycomputer-program products that direct a computer system to perform theactions of the above-described methods and components. Each suchcomputer-program product may comprise sets of instructions (codes)embodied on a computer-readable medium that directs the processor of acomputer system to perform corresponding actions. The instructions maybe configured to run in sequential order, or in parallel (such as underdifferent processing threads), or in a combination thereof. Afterloading the computer-program products on a general purpose computersystem 1726, it is transformed into the special-purpose computer system1700.

Special-purpose computer system 1700 comprises a computer 1702, amonitor 1706 coupled to computer 1702, one or more additional useroutput devices 1730 (optional) coupled to computer 1702, one or moreuser input devices 1740 (e.g., keyboard, mouse, track ball, touchscreen) coupled to computer 1702, an optional communications interface1750 coupled to computer 1702, a computer-program product 1705 stored ina tangible computer-readable memory in computer 1702. Computer-programproduct 1705 directs system 1700 to perform the above-described methods.Computer 1702 may include one or more processors 1760 that communicatewith a number of peripheral devices via a bus subsystem 1790. Theseperipheral devices may include user output device(s) 1730, user inputdevice(s) 1740, communications interface 1750, and a storage subsystem,such as random access memory (RAM) 1770 and non-volatile storage drive1780 (e.g., disk drive, optical drive, solid state drive), which areforms of tangible computer-readable memory.

Computer-program product 1705 may be stored in non-volatile storagedrive 1780 or another computer-readable medium accessible to computer1702 and loaded into memory 1770. Each processor 1760 may comprise amicroprocessor, such as a microprocessor from Intel® or Advanced MicroDevices, Inc.®, or the like. To support computer-program product 1705,the computer 1702 runs an operating system that handles thecommunications of product 1705 with the above-noted components, as wellas the communications between the above-noted components in support ofthe computer-program product 1705. Exemplary operating systems includeWindows® or the like from Microsoft Corporation, Solaris® from SunMicrosystems, LINUX, UNIX, and the like.

User input devices 1740 include all possible types of devices andmechanisms to input information to computer system 1702. These mayinclude a keyboard, a keypad, a mouse, a scanner, a digital drawing pad,a touch screen incorporated into the display, audio input devices suchas voice recognition systems, microphones, and other types of inputdevices. In various embodiments, user input devices 1740 are typicallyembodied as a computer mouse, a trackball, a track pad, a joystick,wireless remote, a drawing tablet, a voice command system. User inputdevices 1740 typically allow a user to select objects, icons, text andthe like that appear on the monitor 1706 via a command such as a clickof a button or the like. User output devices 1730 include all possibletypes of devices and mechanisms to output information from computer1702. These may include a display (e.g., monitor 1706), printers,non-visual displays such as audio output devices, etc.

Communications interface 1750 provides an interface to othercommunication networks and devices and may serve as an interface toreceive data from and transmit data to other systems, WANs and/or theInternet 1618. Embodiments of communications interface 1750 typicallyinclude an Ethernet card, a modem (telephone, satellite, cable, ISDN), a(asynchronous) digital subscriber line (DSL) unit, a FireWire®interface, a USB® interface, a wireless network adapter, and the like.For example, communications interface 1750 may be coupled to a computernetwork, to a FireWire® bus, or the like. In other embodiments,communications interface 1750 may be physically integrated on themotherboard of computer 1602, and/or may be a software program, or thelike.

RAM 1770 and non-volatile storage drive 1780 are examples of tangiblecomputer-readable media configured to store data such ascomputer-program product embodiments of the present invention, includingexecutable computer code, human-readable code, or the like. Other typesof tangible computer-readable media include floppy disks, removable harddisks, optical storage media such as CD-ROMs, DVDs, bar codes,semiconductor memories such as flash memories, read-only-memories(ROMs), battery-backed volatile memories, networked storage devices, andthe like. RAM 1770 and non-volatile storage drive 1780 may be configuredto store the basic programming and data constructs that provide thefunctionality of various embodiments of the present invention, asdescribed above.

Software instruction sets that provide the functionality of the presentinvention may be stored in RAM 1770 and non-volatile storage drive 1780.These instruction sets or code may be executed by the processor(s) 1760.RAM 1770 and non-volatile storage drive 1780 may also provide arepository to store data and data structures used in accordance with thepresent invention. RAM 1770 and non-volatile storage drive 1780 mayinclude a number of memories including a main random access memory (RAM)to store instructions and data during program execution and a read-onlymemory (ROM) in which fixed instructions are stored. RAM 1770 andnon-volatile storage drive 1780 may include a file storage subsystemproviding persistent (non-volatile) storage of program and/or datafiles. RAM 1770 and non-volatile storage drive 1780 may also includeremovable storage systems, such as removable flash memory.

Bus subsystem 1790 provides a mechanism to allow the various componentsand subsystems of computer 1702 to communicate with each other asintended. Although bus subsystem 1790 is shown schematically as a singlebus, alternative embodiments of the bus subsystem may utilize multiplebusses or communication paths within the computer 1702.

What is claimed is:
 1. A smart-home device comprising: a recordingdevice configured to record sound during a first time interval; a memorydevice comprising a plurality of stored sound profiles; a processingsystem configured to: receive a communication transmitted from ahousehold appliance indicating that the household appliance isoperating; in response to receiving the communication transmitted fromthe household appliance, select a stored sound profile from theplurality of stored sound profiles based on a sound made by thehousehold appliance while the household appliance is operating; andperform a noise-cancellation routine on the sound recorded during thefirst time interval, wherein the stored sound profile is used as aninitial background noise profile for the noise-cancelation routine. 2.The smart-home device of claim 1 wherein the smart-home device comprisesa hazard detector, the hazard detector comprising: a smoke detector; anda carbon monoxide detector.
 3. The smart-home device of claim 1 whereinthe smart-home device comprises a thermostat, the thermostat comprising:a temperature sensor; and a motion sensor.
 4. The smart-home device ofclaim 1, further comprising a network interface in communication with aremote server.
 5. The smart-home device of claim 4, wherein theprocessing system is further configured to: receive an environmentalinput; send the environmental input to the remote server through thenetwork interface; receive a second stored sound profile through thenetwork interface from the remote server, wherein the remote serverselected the second stored sound profile based on the environmentalinput; and perform the noise-cancellation routine on sound recordedduring a second time interval, wherein the second stored sound profileis used as an initial background noise profile for thenoise-cancellation routine.
 6. The smart-home device of claim 4, whereinthe processing system is further configured to: record a new soundprofile using a microphone; and send the new sound profile to the remoteserver through the network interface.
 7. The smart-home device of claim4, wherein: the smart-home device is in communication through thenetwork interface with the household appliance within a same home; andthe communication is received from the household appliance through thenetwork interface.
 8. The smart-home device of claim 1, wherein thestored sound profile is recorded by a second smart-home device withinthe same home and transmitted from the second smart-home device to thesmart-home device.
 9. A method of selecting an initial background noisefor a noise-cancelation routine in a smart-home device, the methodcomprising: receiving, by the smart-home device, a communication from ahousehold appliance indicating that the household appliance isoperating; selecting, using a processor on the smart-home device, and inresponse to receiving the communication from the household appliance, astored sound profile from a plurality of stored sound profiles, whereinthe stored sound profile is selected based on a sound made by thehousehold appliance while the household appliance is operating; andperforming the noise-cancelation routine on sound received through amicrophone of the smart-home device, wherein the noise-cancelationroutine is performed using the stored sound profile as an initialbackground noise for the noise-cancelation routine.
 10. The method ofclaim 9, wherein the sound received through the microphone of thesmart-home device has a duration of less than 10 seconds.
 11. The methodof claim 9, wherein the noise-cancellation routine uses the initialbackground noise as a starting point for a convergence algorithm. 12.The method of claim 11, further comprising: determining that a secondstored sound profile will cause the convergence algorithm to convergefaster than the stored sound profile; and providing the second storedsound profile to the noise-cancellation routine.
 13. The method of claim9, further comprising, during a learning interval, building theplurality of stored sound profiles by: detecting a time interval duringwhich a steady-state background noise is present; recording thesteady-state background noise; and recording characteristics of theoccurrence of the steady-state background noise.
 14. A non-transitorymemory device comprising instructions that, when executed by one or moreprocessors, cause the one or more processors to perform operationscomprising: receiving, by the one or more processors, a communicationtransmitted from a household appliance indicating that the householdappliance is operating; selecting, using the one or more processors, andin response to receiving the communication transmitted from thehousehold appliance, a stored sound profile from a plurality of storedsound profiles, wherein the stored sound profile is selected based on asound made by the household appliance while the household appliance isoperating; and performing a noise-cancelation routine on a receivedsound, wherein the noise-cancellation routine is performed using thestored sound profile as an initial background noise for thenoise-cancelation routine.
 15. The non-transitory memory device of claim14, wherein the stored sound profile is associated with sound emitted bya household appliance during operation.
 16. The non-transitory memorydevice of claim 14, wherein the received sound comprises a voicecommand.
 17. The non-transitory memory device of claim 14, wherein thenoise-cancelation routine is performed on the received sound inreal-time.